2024-09-02T11:47:53.804 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-02T11:47:53.820 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-02T11:47:53.916 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689 branch: main description: nvmeof/thrash/{base/install centos_latest clusters/4-gateways-1-initiator conf/{disable-pool-app} gateway-initiator-setup/3-subsys-60-namespace thrashers/nvmeof_thrash workloads/fio} email: null first_in_suite: false job_id: '7886689' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon warn on pool no app: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_FAILED_DAEMON - is in error state - failed cephadm daemon sha1: b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b59673c44bd569f9f3db37f87bced695dec5fcbf workunit: branch: wip-nvmeof-teuthology-v6 sha1: 6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 owner: scheduled_vallariag@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - client.0 - - host.a - mon.a - mgr.x - osd.0 - osd.1 - client.1 - ceph.nvmeof.nvmeof.a - - host.b - mon.b - osd.2 - osd.3 - osd.4 - client.2 - ceph.nvmeof.nvmeof.b - - host.c - mon.c - osd.5 - osd.6 - osd.7 - client.3 - ceph.nvmeof.nvmeof.c - - host.d - client.4 - ceph.nvmeof.nvmeof.d seed: 4612 sha1: b59673c44bd569f9f3db37f87bced695dec5fcbf sleep_before_teardown: 0 suite: nvmeof suite_branch: wip-nvmeof-teuthology-v6 suite_path: /home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa suite_relpath: qa suite_repo: https://github.com/vallariag/ceph.git suite_sha1: 6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqbL/V401iEjLfUh7X9TNPLBxbjAm/L1tb2/7cShnho4uE0BNTuKcLxovmbKc6hvxxlaAfWio+Ea8LXkqfOfjY= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbdTcrSDxUhl+Chx7MiQVelHORJ0HGQcnW8I25aF/+4nrq/Ht+ql5E+S0WuUdGlkg2Cey6mGhVsNxFhZwaVewo= smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/OTzoQhV+Dz8W9PddTgT4b3BkmVu40MCc2mGFGDoaymUGvDp7DzmkFQCdyfIQnC29mqkqgUpbftZdYSjJT8GM= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGHohEqpCngSgEtPTMWo+QAyHHHyz2Qq53w/dqFPlkSz30p+lab8mdv7e2ETaWRIXHYea/OyKdFbEU6Q8AGhNu0= smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkszDyJATNJqk1yB6bGiGjGOJtqTTGzaTLKLuBXBswPGKjnNnE2+E5qQj9A65PMYkcKHCPAetNteqiSnmosND0= tasks: - install: extra_packages: - nvme-cli - cephadm: watchdog_setup: null - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch host ls - ceph orch device ls - ceph osd lspools - nvmeof: gateway_config: cli_image: quay.io/ceph/nvmeof-cli:1.2 namespaces_count: 20 subsystems_count: 3 gw_image: quay.io/ceph/nvmeof:1.2 installer: host.a rbd: image_name_prefix: myimage pool_name: mypool - cephadm.wait_for_service: service: nvmeof.mypool.mygroup0 - workunit: clients: client.0: - nvmeof/setup_subsystem.sh - nvmeof/basic_tests.sh env: RBD_IMAGE_PREFIX: myimage RBD_POOL: mypool no_coverage_and_limits: true - nvmeof.thrash: checker_host: client.0 - workunit: clients: client.3: - nvmeof/fio_test.sh --rbd_iostat env: IOSTAT_INTERVAL: '10' RBD_POOL: mypool RUNTIME: '600' no_coverage_and_limits: true timeout: 30m teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-02_11:40:02 tube: smithi use_shaman: true user: vallariag verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-02T11:47:53.916 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa; will attempt to use it 2024-09-02T11:47:53.917 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa/tasks 2024-09-02T11:47:53.917 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-02T11:47:53.919 INFO:teuthology.task.internal:Checking packages... 2024-09-02T11:47:53.940 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b59673c44bd569f9f3db37f87bced695dec5fcbf' 2024-09-02T11:47:53.940 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T11:47:53.940 INFO:teuthology.packaging:ref: None 2024-09-02T11:47:53.940 INFO:teuthology.packaging:tag: None 2024-09-02T11:47:53.940 INFO:teuthology.packaging:branch: main 2024-09-02T11:47:53.940 INFO:teuthology.packaging:sha1: b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:47:53.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-02T11:47:54.199 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4637.g42c1c3d4 2024-09-02T11:47:54.201 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-02T11:47:54.222 INFO:teuthology.task.internal:no buildpackages task found 2024-09-02T11:47:54.222 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-02T11:47:54.239 INFO:teuthology.task.internal:Saving configuration 2024-09-02T11:47:54.252 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-02T11:47:54.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-02T11:47:54.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 11:42:51.939078', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqbL/V401iEjLfUh7X9TNPLBxbjAm/L1tb2/7cShnho4uE0BNTuKcLxovmbKc6hvxxlaAfWio+Ea8LXkqfOfjY='} 2024-09-02T11:47:54.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 11:42:51.941822', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbdTcrSDxUhl+Chx7MiQVelHORJ0HGQcnW8I25aF/+4nrq/Ht+ql5E+S0WuUdGlkg2Cey6mGhVsNxFhZwaVewo='} 2024-09-02T11:47:54.356 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 11:42:51.943159', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/OTzoQhV+Dz8W9PddTgT4b3BkmVu40MCc2mGFGDoaymUGvDp7DzmkFQCdyfIQnC29mqkqgUpbftZdYSjJT8GM='} 2024-09-02T11:47:54.387 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 11:42:51.937488', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGHohEqpCngSgEtPTMWo+QAyHHHyz2Qq53w/dqFPlkSz30p+lab8mdv7e2ETaWRIXHYea/OyKdFbEU6Q8AGhNu0='} 2024-09-02T11:47:54.423 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 11:42:51.940454', 'locked_by': 'scheduled_vallariag@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkszDyJATNJqk1yB6bGiGjGOJtqTTGzaTLKLuBXBswPGKjnNnE2+E5qQj9A65PMYkcKHCPAetNteqiSnmosND0='} 2024-09-02T11:47:54.423 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-02T11:47:54.430 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['client.0'] 2024-09-02T11:47:54.430 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['host.a', 'mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.1', 'ceph.nvmeof.nvmeof.a'] 2024-09-02T11:47:54.430 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['host.b', 'mon.b', 'osd.2', 'osd.3', 'osd.4', 'client.2', 'ceph.nvmeof.nvmeof.b'] 2024-09-02T11:47:54.430 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['host.c', 'mon.c', 'osd.5', 'osd.6', 'osd.7', 'client.3', 'ceph.nvmeof.nvmeof.c'] 2024-09-02T11:47:54.430 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['host.d', 'client.4', 'ceph.nvmeof.nvmeof.d'] 2024-09-02T11:47:54.430 INFO:teuthology.run_tasks:Running task console_log... 2024-09-02T11:47:54.613 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f49b1cfbac0>, signals=[15]) 2024-09-02T11:47:54.613 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-02T11:47:54.620 INFO:teuthology.task.internal:Opening connections... 2024-09-02T11:47:54.620 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-09-02T11:47:54.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:47:54.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-09-02T11:47:54.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:47:54.795 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-09-02T11:47:54.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:47:54.883 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2024-09-02T11:47:54.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:47:54.968 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-09-02T11:47:54.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:47:55.052 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-02T11:47:55.059 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-09-02T11:47:55.078 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-09-02T11:47:55.078 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-09-02T11:47:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-09-02T11:47:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T11:47:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-09-02T11:47:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T11:47:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T11:47:55.137 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T11:47:55.137 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-09-02T11:47:55.166 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-09-02T11:47:55.184 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-09-02T11:47:55.184 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-09-02T11:47:55.241 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-09-02T11:47:55.241 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-09-02T11:47:55.241 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T11:47:55.242 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T11:47:55.243 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-09-02T11:47:55.274 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-09-02T11:47:55.293 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-09-02T11:47:55.294 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-09-02T11:47:55.352 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-09-02T11:47:55.352 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-09-02T11:47:55.352 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T11:47:55.353 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T11:47:55.354 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-09-02T11:47:55.383 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2024-09-02T11:47:55.402 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2024-09-02T11:47:55.403 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2024-09-02T11:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2024-09-02T11:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="9" 2024-09-02T11:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2024-09-02T11:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2024-09-02T11:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="9" 2024-09-02T11:47:55.460 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el9" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:LOGO="fedora-logo-icon" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T11:47:55.461 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T11:47:55.462 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2024-09-02T11:47:55.491 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-09-02T11:47:55.510 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-09-02T11:47:55.510 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-09-02T11:47:55.568 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2024-09-02T11:47:55.568 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="9" 2024-09-02T11:47:55.568 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2024-09-02T11:47:55.568 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="9" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el9" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:LOGO="fedora-logo-icon" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T11:47:55.569 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T11:47:55.569 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-09-02T11:47:55.599 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-02T11:47:55.609 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-02T11:47:55.646 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-02T11:47:55.646 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-09-02T11:47:55.649 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-09-02T11:47:55.651 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-09-02T11:47:55.653 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2024-09-02T11:47:55.656 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-09-02T11:47:55.672 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-02T11:47:55.679 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-02T11:47:55.679 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-09-02T11:47:55.706 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-09-02T11:47:55.709 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-09-02T11:47:55.711 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2024-09-02T11:47:55.713 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-09-02T11:47:55.822 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-02T11:47:55.989 INFO:teuthology.run_tasks:Running task kernel... 2024-09-02T11:47:56.007 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-02T11:47:56.007 INFO:teuthology.task.kernel:config {'client.0': {'kdb': 1, 'sha1': 'distro'}, 'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}, 'host.c': {'kdb': 1, 'sha1': 'distro'}, 'host.d': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-02T11:47:56.008 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T11:47:56.008 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T11:47:56.009 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T11:47:56.009 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T11:47:56.010 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T11:47:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:47:56.027 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2024-09-02T11:47:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:47:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:47:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:47:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:47:56.030 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-09-02T11:47:56.031 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-09-02T11:47:56.031 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-02T11:47:56.031 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-09-02T11:47:56.085 INFO:teuthology.orchestra.run.smithi094.stdout:5.14.0-503.el9.x86_64 2024-09-02T11:47:56.085 INFO:teuthology.task.kernel:Running kernel on smithi094: 5.14.0-503.el9.x86_64 2024-09-02T11:47:56.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2024-09-02T11:47:56.087 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-503.el9.x86_64 2024-09-02T11:47:56.087 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-503.el9.x86_64 2024-09-02T11:47:56.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-09-02T11:47:56.090 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-503.el9.x86_64 2024-09-02T11:47:56.090 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-503.el9.x86_64 2024-09-02T11:47:56.090 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-503.el9.x86_64 2024-09-02T11:47:56.090 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-503.el9.x86_64 2024-09-02T11:47:56.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-09-02T11:47:56.090 INFO:teuthology.task.kernel:Running kernel on smithi099: 5.14.0-503.el9.x86_64 2024-09-02T11:47:56.091 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-09-02T11:47:56.091 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-503.el9.x86_64 2024-09-02T11:47:56.091 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-09-02T11:47:58.362 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-02T11:47:59.043 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.2 MB 00:01 2024-09-02T11:47:59.516 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-02T11:47:59.734 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-09-02T11:47:59.874 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-02T11:48:02.793 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-02T11:48:03.955 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-02T11:48:04.408 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 20 MB 00:03 2024-09-02T11:48:08.001 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-02T11:48:08.119 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.6 MB 00:01 2024-09-02T11:48:08.297 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.6 MB 00:00 2024-09-02T11:48:08.516 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-02T11:48:09.334 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.6 MB 00:01 2024-09-02T11:48:09.603 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 18 kB 00:00 2024-09-02T11:48:09.806 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 18 kB 00:00 2024-09-02T11:48:10.607 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-02T11:48:10.768 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 18 kB 00:00 2024-09-02T11:48:10.835 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-02T11:48:11.689 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-02T11:48:13.017 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.6 MB 00:01 2024-09-02T11:48:14.401 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 104 kB/s | 18 kB 00:00 2024-09-02T11:48:14.916 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.6 MB 00:02 2024-09-02T11:48:15.298 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-02T11:48:15.979 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-02T11:48:16.408 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-09-02T11:48:16.634 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-09-02T11:48:17.128 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-09-02T11:48:17.463 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-02T11:48:17.986 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T11:48:17.986 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T11:48:18.074 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:48:18.075 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:48:18.075 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:48:18.167 DEBUG:teuthology.orchestra.run.smithi087:> echo no | sudo yum reinstall kernel || true 2024-09-02T11:48:18.514 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T11:48:18.515 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T11:48:18.602 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:48:18.603 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:48:18.603 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:48:18.705 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 11:48:15 AM UTC. 2024-09-02T11:48:18.716 DEBUG:teuthology.orchestra.run.smithi038:> echo no | sudo yum reinstall kernel || true 2024-09-02T11:48:18.836 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:48:18.843 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:48:18.843 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:18.843 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:48:18.843 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalling: 2024-09-02T11:48:18.843 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:18.843 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:48:18.844 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:48:18.844 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:48:18.844 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:48:18.844 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2024-09-02T11:48:18.844 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 0 2024-09-02T11:48:18.844 INFO:teuthology.orchestra.run.smithi087.stderr:Operation aborted. 2024-09-02T11:48:18.901 INFO:teuthology.orchestra.run.smithi087.stdout:Is this ok [y/N]: 2024-09-02T11:48:18.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum reinstall -y kernel || true 2024-09-02T11:48:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T11:48:19.168 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T11:48:19.244 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:48:19.244 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:48:19.245 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:48:19.268 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 11:48:16 AM UTC. 2024-09-02T11:48:19.367 DEBUG:teuthology.orchestra.run.smithi094:> echo no | sudo yum reinstall kernel || true 2024-09-02T11:48:19.417 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:48:19.425 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:48:19.426 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:48:19.426 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:48:19.426 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.9 M 2024-09-02T11:48:19.426 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-09-02T11:48:19.426 INFO:teuthology.orchestra.run.smithi038.stderr:Operation aborted. 2024-09-02T11:48:19.435 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:04 ago on Mon 02 Sep 2024 11:48:15 AM UTC. 2024-09-02T11:48:19.483 INFO:teuthology.orchestra.run.smithi038.stdout:Is this ok [y/N]: 2024-09-02T11:48:19.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum reinstall -y kernel || true 2024-09-02T11:48:19.560 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalling: 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:48:19.561 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2024-09-02T11:48:19.562 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 0 2024-09-02T11:48:19.562 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:48:19.900 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:02 ago on Mon 02 Sep 2024 11:48:17 AM UTC. 2024-09-02T11:48:20.033 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:04 ago on Mon 02 Sep 2024 11:48:16 AM UTC. 2024-09-02T11:48:20.045 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:48:20.052 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:48:20.052 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:20.052 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:48:20.052 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalling: 2024-09-02T11:48:20.052 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:20.052 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:48:20.053 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:48:20.053 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:48:20.053 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:48:20.053 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2024-09-02T11:48:20.053 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 0 2024-09-02T11:48:20.053 INFO:teuthology.orchestra.run.smithi094.stderr:Operation aborted. 2024-09-02T11:48:20.116 INFO:teuthology.orchestra.run.smithi094.stdout:Is this ok [y/N]: 2024-09-02T11:48:20.116 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum reinstall -y kernel || true 2024-09-02T11:48:20.165 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalling: 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:48:20.166 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:48:20.167 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:48:20.167 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 1.9 M 2024-09-02T11:48:20.167 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 0 2024-09-02T11:48:20.167 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:48:20.645 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 11:48:17 AM UTC. 2024-09-02T11:48:20.669 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-503.el9.x86_64.rpm 3.9 MB/s | 1.9 MB 00:00 2024-09-02T11:48:20.669 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:48:20.669 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.7 MB/s | 1.9 MB 00:00 2024-09-02T11:48:20.698 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:48:20.745 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-02T11:48:20.771 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:48:20.771 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalling: 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:48:20.772 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:48:20.773 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 1.9 M 2024-09-02T11:48:20.773 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 0 2024-09-02T11:48:20.773 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:48:20.927 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:48:20.927 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:48:20.959 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:48:20.959 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:48:21.112 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:48:21.187 INFO:teuthology.orchestra.run.smithi038.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:21.188 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:21.282 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-503.el9.x86_64.rpm 1.1 MB/s | 1.9 MB 00:01 2024-09-02T11:48:21.282 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:48:21.282 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-09-02T11:48:21.313 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:48:21.463 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-503.el9.x86_64.rpm 3.2 MB/s | 1.9 MB 00:00 2024-09-02T11:48:21.464 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:48:21.464 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.7 MB/s | 1.9 MB 00:00 2024-09-02T11:48:21.501 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:48:21.551 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:48:21.551 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:48:21.582 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:48:21.582 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:48:21.745 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:48:21.746 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:48:21.746 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:48:21.777 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:48:21.777 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:48:21.842 INFO:teuthology.orchestra.run.smithi087.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:21.842 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:21.936 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:48:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:22.032 INFO:teuthology.orchestra.run.smithi094.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:22.032 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:22.615 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:22.616 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:22.775 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:22.775 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:22.792 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T11:48:22.792 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T11:48:22.812 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:22.812 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:48:22.812 INFO:teuthology.orchestra.run.smithi038.stdout:Reinstalled: 2024-09-02T11:48:22.812 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:22.812 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:48:22.812 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:48:22.871 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:48:22.872 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:48:22.872 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:48:22.978 DEBUG:teuthology.orchestra.run.smithi099:> echo no | sudo yum reinstall kernel || true 2024-09-02T11:48:23.001 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T11:48:23.002 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-09-02T11:48:23.059 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:23.060 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T11:48:23.060 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T11:48:23.060 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-02T11:48:23.060 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T11:48:23.092 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-09-02T11:48:23.114 DEBUG:teuthology.parallel:result is None 2024-09-02T11:48:23.338 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:23.338 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:48:23.338 INFO:teuthology.orchestra.run.smithi087.stdout:Reinstalled: 2024-09-02T11:48:23.338 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:23.338 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:48:23.338 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:48:23.513 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T11:48:23.514 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:23.514 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:48:23.514 INFO:teuthology.orchestra.run.smithi094.stdout:Reinstalled: 2024-09-02T11:48:23.515 INFO:teuthology.orchestra.run.smithi094.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:23.515 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:48:23.515 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:48:23.557 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 11:48:20 AM UTC. 2024-09-02T11:48:23.571 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:23.572 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T11:48:23.572 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T11:48:23.572 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-09-02T11:48:23.572 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T11:48:23.601 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-09-02T11:48:23.628 DEBUG:teuthology.parallel:result is None 2024-09-02T11:48:23.700 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T11:48:23.702 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:48:23.709 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:48:23.709 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:23.709 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout:Reinstalling: 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.9 M 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 0 2024-09-02T11:48:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:Operation aborted. 2024-09-02T11:48:23.759 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:23.759 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T11:48:23.759 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T11:48:23.759 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2024-09-02T11:48:23.759 DEBUG:teuthology.orchestra.run.smithi094:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T11:48:23.775 INFO:teuthology.orchestra.run.smithi099.stdout:Is this ok [y/N]: 2024-09-02T11:48:23.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum reinstall -y kernel || true 2024-09-02T11:48:23.787 INFO:teuthology.orchestra.run.smithi094.stdout:ttyS1 2024-09-02T11:48:23.810 DEBUG:teuthology.parallel:result is None 2024-09-02T11:48:24.329 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:04 ago on Mon 02 Sep 2024 11:48:20 AM UTC. 2024-09-02T11:48:24.467 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:48:24.468 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:48:24.468 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:24.468 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout:Reinstalling: 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.9 M 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 0 2024-09-02T11:48:24.469 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:48:25.084 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T11:48:25.084 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T11:48:25.160 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:48:25.161 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:48:25.161 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:48:25.188 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-503.el9.x86_64.rpm 4.1 MB/s | 1.9 MB 00:00 2024-09-02T11:48:25.188 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:48:25.188 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2024-09-02T11:48:25.217 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:48:25.275 DEBUG:teuthology.orchestra.run.smithi064:> echo no | sudo yum reinstall kernel || true 2024-09-02T11:48:25.461 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:48:25.461 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:48:25.492 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:48:25.493 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:48:25.674 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:48:25.765 INFO:teuthology.orchestra.run.smithi099.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:25.766 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:25.823 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 11:48:22 AM UTC. 2024-09-02T11:48:25.977 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:48:25.986 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:48:25.987 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.9 M 2024-09-02T11:48:25.987 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-09-02T11:48:25.987 INFO:teuthology.orchestra.run.smithi064.stderr:Operation aborted. 2024-09-02T11:48:26.041 INFO:teuthology.orchestra.run.smithi064.stdout:Is this ok [y/N]: 2024-09-02T11:48:26.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum reinstall -y kernel || true 2024-09-02T11:48:26.559 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:26.559 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:26.588 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:04 ago on Mon 02 Sep 2024 11:48:22 AM UTC. 2024-09-02T11:48:26.723 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalling: 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:48:26.724 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:48:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:48:26.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:48:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.9 M 2024-09-02T11:48:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 0 2024-09-02T11:48:26.725 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:48:27.096 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-503.el9.x86_64.rpm 8.4 MB/s | 1.9 MB 00:00 2024-09-02T11:48:27.096 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:48:27.096 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.0 MB/s | 1.9 MB 00:00 2024-09-02T11:48:27.128 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:48:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:48:27.229 INFO:teuthology.orchestra.run.smithi099.stdout:Reinstalled: 2024-09-02T11:48:27.229 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:27.230 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:48:27.230 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:48:27.367 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:48:27.367 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:48:27.399 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:48:27.399 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:48:27.443 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T11:48:27.501 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:27.502 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T11:48:27.502 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T11:48:27.502 INFO:teuthology.task.kernel:Enabling kdb on host.d... 2024-09-02T11:48:27.502 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T11:48:27.532 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-09-02T11:48:27.557 DEBUG:teuthology.parallel:result is None 2024-09-02T11:48:27.565 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:48:27.648 INFO:teuthology.orchestra.run.smithi064.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:27.648 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:28.432 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:28.432 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T11:48:29.161 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T11:48:29.161 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:48:29.161 INFO:teuthology.orchestra.run.smithi064.stdout:Reinstalled: 2024-09-02T11:48:29.161 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:29.161 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:48:29.162 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:48:29.358 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T11:48:29.417 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T11:48:29.417 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T11:48:29.418 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T11:48:29.418 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-09-02T11:48:29.418 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T11:48:29.448 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-09-02T11:48:29.466 DEBUG:teuthology.parallel:result is None 2024-09-02T11:48:29.466 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-02T11:48:29.473 INFO:teuthology.task.internal:Creating test directory... 2024-09-02T11:48:29.474 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T11:48:29.476 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T11:48:29.508 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T11:48:29.511 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T11:48:29.513 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T11:48:29.532 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-02T11:48:29.540 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-02T11:48:29.576 INFO:teuthology.task.internal:Creating archive directory... 2024-09-02T11:48:29.576 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T11:48:29.578 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T11:48:29.581 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T11:48:29.583 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T11:48:29.585 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T11:48:29.627 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-02T11:48:29.634 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-02T11:48:29.635 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T11:48:29.642 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T11:48:29.651 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T11:48:29.654 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T11:48:29.663 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T11:48:29.685 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.696 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.709 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.715 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.719 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.727 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.734 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.747 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.759 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.772 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T11:48:29.774 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-02T11:48:29.781 INFO:teuthology.task.internal:Configuring sudo... 2024-09-02T11:48:29.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T11:48:29.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T11:48:29.785 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T11:48:29.788 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T11:48:29.792 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T11:48:29.821 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-02T11:48:29.830 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-02T11:48:29.830 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T11:48:29.852 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T11:48:29.854 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T11:48:29.857 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T11:48:29.860 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T11:48:29.879 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:29.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:29.999 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.129 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:48:30.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T11:48:30.196 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:30.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:30.308 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.388 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:48:30.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T11:48:30.453 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:30.508 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:30.561 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.649 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:48:30.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T11:48:30.715 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:30.768 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:30.819 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.904 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:30.947 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:48:30.947 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T11:48:31.014 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:31.065 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T11:48:31.114 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:31.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T11:48:31.241 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:48:31.241 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T11:48:31.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-09-02T11:48:31.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-09-02T11:48:31.314 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-09-02T11:48:31.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2024-09-02T11:48:31.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-09-02T11:48:31.351 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:48:31.353 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:48:31.358 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:48:31.364 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:48:31.391 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:48:31.800 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-02T11:48:31.807 INFO:teuthology.task.internal:Starting timer... 2024-09-02T11:48:31.807 INFO:teuthology.run_tasks:Running task pcp... 2024-09-02T11:48:31.862 INFO:teuthology.run_tasks:Running task selinux... 2024-09-02T11:48:31.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-09-02T11:48:31.966 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-09-02T11:48:31.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-09-02T11:48:32.012 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-09-02T11:48:32.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-09-02T11:48:32.062 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-09-02T11:48:32.064 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2024-09-02T11:48:32.120 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2024-09-02T11:48:32.122 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-09-02T11:48:32.175 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-09-02T11:48:32.177 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-02T11:48:32.177 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-09-02T11:48:32.199 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-09-02T11:48:32.199 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-09-02T11:48:32.224 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-09-02T11:48:32.224 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-09-02T11:48:32.251 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-09-02T11:48:32.251 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2024-09-02T11:48:32.274 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2024-09-02T11:48:32.274 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-09-02T11:48:32.294 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-09-02T11:48:32.294 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi099.front.sepia.ceph.com': 'permissive'} 2024-09-02T11:48:32.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T11:48:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:48:32.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T11:48:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:48:32.346 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T11:48:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:48:32.373 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T11:48:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:48:32.400 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T11:48:32.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:48:32.425 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-02T11:48:32.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-09-02T11:48:32.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-09-02T11:48:32.489 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-09-02T11:48:32.520 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2024-09-02T11:48:32.554 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-09-02T11:48:32.580 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-02T11:48:32.590 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-02T11:48:32.601 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-02T11:48:32.602 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi064.front.sepia.ceph.com,smithi087.front.sepia.ceph.com,smithi094.front.sepia.ceph.com,smithi099.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-02T11:50:24.987 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com'), Remote(name='ubuntu@smithi099.front.sepia.ceph.com')] 2024-09-02T11:50:24.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-02T11:50:24.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:50:25.078 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-09-02T11:50:25.167 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-02T11:50:25.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-02T11:50:25.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:50:25.251 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-02T11:50:25.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-02T11:50:25.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-02T11:50:25.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:50:25.420 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-09-02T11:50:25.510 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-02T11:50:25.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-02T11:50:25.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:50:25.596 DEBUG:teuthology.orchestra.run.smithi094:> true 2024-09-02T11:50:25.677 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2024-09-02T11:50:25.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-02T11:50:25.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T11:50:25.764 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-09-02T11:50:25.842 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-02T11:50:25.842 INFO:teuthology.run_tasks:Running task clock... 2024-09-02T11:50:25.852 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-02T11:50:25.852 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T11:50:25.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:50:25.855 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T11:50:25.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:50:25.857 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T11:50:25.858 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:50:25.860 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T11:50:25.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:50:25.862 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T11:50:25.862 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:50:25.886 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T11:50:25.889 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T11:50:25.892 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T11:50:25.894 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T11:50:25.901 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T11:50:25.905 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T11:50:25.908 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T11:50:25.910 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T11:50:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T11:50:25.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-09-02T11:50:25.930 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T11:50:25.932 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-09-02T11:50:25.934 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-09-02T11:50:25.937 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2024-09-02T11:50:25.940 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-09-02T11:50:25.944 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-09-02T11:50:25.946 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-09-02T11:50:25.948 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2024-09-02T11:50:25.955 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T11:50:25.955 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-09-02T11:50:25.959 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T11:50:25.961 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T11:50:25.963 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T11:50:25.967 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-09-02T11:50:25.970 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T11:50:25.973 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T11:50:25.975 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T11:50:25.978 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T11:50:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T11:50:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T11:50:26.030 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:50:26.031 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:50:26.033 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:50:26.033 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-02T11:50:26.034 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.034 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.034 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.034 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.035 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:50:26.035 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-09-02T11:50:26.035 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.035 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.035 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.036 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.036 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:50:26.039 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:50:26.040 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:50:26.040 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-09-02T11:50:26.040 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.040 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.040 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.041 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.043 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:50:26.043 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-09-02T11:50:26.043 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.043 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.043 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.043 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:50:26.053 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:50:26.053 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-09-02T11:50:26.054 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.054 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.054 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.054 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:50:26.055 INFO:teuthology.run_tasks:Running task install... 2024-09-02T11:50:26.063 DEBUG:teuthology.task.install:project ceph 2024-09-02T11:50:26.063 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf'}} 2024-09-02T11:50:26.063 DEBUG:teuthology.task.install:config {'extra_packages': ['nvme-cli'], 'flavor': 'default', 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf'} 2024-09-02T11:50:26.063 INFO:teuthology.task.install:Using flavor: default 2024-09-02T11:50:26.069 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-02T11:50:26.069 INFO:teuthology.task.install:extra packages: ['nvme-cli'] 2024-09-02T11:50:26.069 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf', 'tag': None, 'wait_for_package': False} 2024-09-02T11:50:26.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:50:26.072 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf', 'tag': None, 'wait_for_package': False} 2024-09-02T11:50:26.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:50:26.073 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf', 'tag': None, 'wait_for_package': False} 2024-09-02T11:50:26.073 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:50:26.074 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf', 'tag': None, 'wait_for_package': False} 2024-09-02T11:50:26.074 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:50:26.075 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['nvme-cli'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf', 'tag': None, 'wait_for_package': False} 2024-09-02T11:50:26.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:50:26.248 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/ 2024-09-02T11:50:26.249 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4585.gb59673c4 2024-09-02T11:50:26.261 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/ 2024-09-02T11:50:26.261 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4585.gb59673c4 2024-09-02T11:50:26.262 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/ 2024-09-02T11:50:26.262 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4585.gb59673c4 2024-09-02T11:50:26.289 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/ 2024-09-02T11:50:26.289 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4585.gb59673c4 2024-09-02T11:50:26.290 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/ 2024-09-02T11:50:26.290 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4585.gb59673c4 2024-09-02T11:50:26.385 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T11:50:26.386 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:50:26.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T11:50:26.392 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T11:50:26.392 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:50:26.392 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T11:50:26.397 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T11:50:26.397 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:50:26.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T11:50:26.410 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T11:50:26.410 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:50:26.410 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T11:50:26.413 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T11:50:26.413 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:50:26.413 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T11:50:26.415 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2024-09-02T11:50:26.415 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b59673c44bd569f9f3db37f87bced695dec5fcbf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T11:50:26.419 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2024-09-02T11:50:26.419 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b59673c44bd569f9f3db37f87bced695dec5fcbf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T11:50:26.424 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2024-09-02T11:50:26.424 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b59673c44bd569f9f3db37f87bced695dec5fcbf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T11:50:26.436 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2024-09-02T11:50:26.436 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b59673c44bd569f9f3db37f87bced695dec5fcbf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T11:50:26.441 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, nvme-cli on remote rpm x86_64 2024-09-02T11:50:26.441 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b59673c44bd569f9f3db37f87bced695dec5fcbf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T11:50:26.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T11:50:26.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T11:50:26.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T11:50:26.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T11:50:26.507 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T11:50:26.558 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T11:50:26.560 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T11:50:26.566 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T11:50:26.577 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T11:50:26.582 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T11:50:26.622 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-09-02T11:50:26.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-09-02T11:50:26.625 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2024-09-02T11:50:26.626 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2024-09-02T11:50:26.631 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-09-02T11:50:26.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-09-02T11:50:26.641 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2024-09-02T11:50:26.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2024-09-02T11:50:26.648 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-09-02T11:50:26.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-09-02T11:50:26.851 INFO:teuthology.orchestra.run.smithi094.stdout:45 files removed 2024-09-02T11:50:26.855 INFO:teuthology.orchestra.run.smithi038.stdout:45 files removed 2024-09-02T11:50:26.863 INFO:teuthology.orchestra.run.smithi064.stdout:45 files removed 2024-09-02T11:50:26.867 INFO:teuthology.orchestra.run.smithi099.stdout:45 files removed 2024-09-02T11:50:26.876 INFO:teuthology.orchestra.run.smithi087.stdout:45 files removed 2024-09-02T11:50:26.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw 2024-09-02T11:50:26.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-09-02T11:50:26.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-09-02T11:50:26.894 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2024-09-02T11:50:26.904 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-09-02T11:50:27.562 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 206 kB/s | 84 kB 00:00 2024-09-02T11:50:27.568 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 208 kB/s | 84 kB 00:00 2024-09-02T11:50:27.573 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 207 kB/s | 84 kB 00:00 2024-09-02T11:50:27.613 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 190 kB/s | 84 kB 00:00 2024-09-02T11:50:27.621 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 190 kB/s | 84 kB 00:00 2024-09-02T11:50:28.079 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 51 kB/s | 24 kB 00:00 2024-09-02T11:50:28.086 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 51 kB/s | 24 kB 00:00 2024-09-02T11:50:28.090 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 50 kB/s | 24 kB 00:00 2024-09-02T11:50:28.113 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 53 kB/s | 24 kB 00:00 2024-09-02T11:50:28.130 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 51 kB/s | 24 kB 00:00 2024-09-02T11:50:28.361 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-02T11:50:28.363 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-02T11:50:28.366 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-02T11:50:28.389 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-02T11:50:28.540 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-09-02T11:50:28.878 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-02T11:50:28.921 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-02T11:50:28.974 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-02T11:50:30.581 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.2 MB 00:02 2024-09-02T11:50:30.889 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-02T11:50:31.164 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.2 MB 00:02 2024-09-02T11:50:34.258 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-02T11:50:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-02T11:50:35.515 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.6 MB 00:00 2024-09-02T11:50:36.924 INFO:teuthology.orchestra.run.smithi094.stdout:CentOS Stream 9 - Extras packages 95 kB/s | 18 kB 00:00 2024-09-02T11:50:37.691 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-02T11:50:38.187 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-02T11:50:38.667 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.6 MB 00:00 2024-09-02T11:50:40.447 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.6 MB 00:02 2024-09-02T11:50:40.534 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 18 kB 00:00 2024-09-02T11:50:41.705 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-02T11:50:42.155 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 18 kB 00:00 2024-09-02T11:50:42.908 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.6 MB 00:01 2024-09-02T11:50:43.068 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-02T11:50:44.626 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 1.7 kB/s | 1.7 kB 00:01 2024-09-02T11:50:44.892 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 18 kB 00:00 2024-09-02T11:50:45.905 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-02T11:50:46.969 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:50:46.971 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:50:46.971 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:50:46.971 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:50:46.971 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:50:46.971 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 11 M 2024-09-02T11:50:46.971 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.5 M 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 22 M 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 25 k 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T11:50:46.972 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 755 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 508 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 45 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 143 k 2024-09-02T11:50:46.973 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 162 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 320 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 305 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 99 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout:Installing weak dependencies: 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T11:50:46.974 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout:Install 30 Packages 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 2 Packages 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 62 M 2024-09-02T11:50:46.975 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:50:47.144 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-02T11:50:47.868 INFO:teuthology.orchestra.run.smithi094.stdout:(1/32): ceph-base-19.3.0-4585.gb59673c4.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-02T11:50:47.927 INFO:teuthology.orchestra.run.smithi094.stdout:(2/32): ceph-selinux-19.3.0-4585.gb59673c4.el9. 432 kB/s | 25 kB 00:00 2024-09-02T11:50:48.034 INFO:teuthology.orchestra.run.smithi094.stdout:(3/32): libcephfs2-19.3.0-4585.gb59673c4.el9.x8 6.9 MB/s | 755 kB 00:00 2024-09-02T11:50:48.095 INFO:teuthology.orchestra.run.smithi094.stdout:(4/32): libradosstriper1-19.3.0-4585.gb59673c4. 8.3 MB/s | 508 kB 00:00 2024-09-02T11:50:48.387 INFO:teuthology.orchestra.run.smithi094.stdout:(5/32): ceph-radosgw-19.3.0-4585.gb59673c4.el9. 10 MB/s | 11 MB 00:01 2024-09-02T11:50:48.487 INFO:teuthology.orchestra.run.smithi094.stdout:(6/32): librgw2-19.3.0-4585.gb59673c4.el9.x86_6 14 MB/s | 5.5 MB 00:00 2024-09-02T11:50:48.512 INFO:teuthology.orchestra.run.smithi094.stdout:(7/32): python3-ceph-argparse-19.3.0-4585.gb596 361 kB/s | 45 kB 00:00 2024-09-02T11:50:48.537 INFO:teuthology.orchestra.run.smithi094.stdout:(8/32): python3-ceph-common-19.3.0-4585.gb59673 2.8 MB/s | 143 kB 00:00 2024-09-02T11:50:48.562 INFO:teuthology.orchestra.run.smithi094.stdout:(9/32): python3-cephfs-19.3.0-4585.gb59673c4.el 3.2 MB/s | 162 kB 00:00 2024-09-02T11:50:48.599 INFO:teuthology.orchestra.run.smithi094.stdout:(10/32): python3-rados-19.3.0-4585.gb59673c4.el 5.1 MB/s | 320 kB 00:00 2024-09-02T11:50:48.629 INFO:teuthology.orchestra.run.smithi094.stdout:(11/32): python3-rbd-19.3.0-4585.gb59673c4.el9. 4.5 MB/s | 305 kB 00:00 2024-09-02T11:50:48.655 INFO:teuthology.orchestra.run.smithi094.stdout:(12/32): python3-rgw-19.3.0-4585.gb59673c4.el9. 1.8 MB/s | 99 kB 00:00 2024-09-02T11:50:48.707 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-02T11:50:48.864 INFO:teuthology.orchestra.run.smithi094.stdout:(13/32): ceph-common-19.3.0-4585.gb59673c4.el9. 14 MB/s | 22 MB 00:01 2024-09-02T11:50:48.905 INFO:teuthology.orchestra.run.smithi094.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-09-02T11:50:48.947 INFO:teuthology.orchestra.run.smithi094.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 544 kB/s | 45 kB 00:00 2024-09-02T11:50:48.972 INFO:teuthology.orchestra.run.smithi094.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 505 kB/s | 160 kB 00:00 2024-09-02T11:50:49.006 INFO:teuthology.orchestra.run.smithi094.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 42 kB 00:00 2024-09-02T11:50:49.031 INFO:teuthology.orchestra.run.smithi094.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-02T11:50:49.057 INFO:teuthology.orchestra.run.smithi094.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-02T11:50:49.073 INFO:teuthology.orchestra.run.smithi094.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 860 kB/s | 36 kB 00:00 2024-09-02T11:50:49.119 INFO:teuthology.orchestra.run.smithi094.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 772 kB/s | 47 kB 00:00 2024-09-02T11:50:49.149 INFO:teuthology.orchestra.run.smithi094.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-02T11:50:49.208 INFO:teuthology.orchestra.run.smithi094.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.3 MB/s | 309 kB 00:00 2024-09-02T11:50:49.224 INFO:teuthology.orchestra.run.smithi094.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 3.3 MB/s | 49 kB 00:00 2024-09-02T11:50:49.240 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-09-02T11:50:49.241 INFO:teuthology.orchestra.run.smithi094.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 337 kB/s | 26 kB 00:00 2024-09-02T11:50:49.267 INFO:teuthology.orchestra.run.smithi094.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-02T11:50:49.300 INFO:teuthology.orchestra.run.smithi094.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-02T11:50:49.359 INFO:teuthology.orchestra.run.smithi094.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-02T11:50:49.426 INFO:teuthology.orchestra.run.smithi094.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-02T11:50:49.496 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 11 M 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.5 M 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:49.499 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 22 M 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 25 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 755 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 508 k 2024-09-02T11:50:49.500 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 45 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 143 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 162 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 320 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 305 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 99 k 2024-09-02T11:50:49.501 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-09-02T11:50:49.502 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-09-02T11:50:49.503 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:50:49.503 INFO:teuthology.orchestra.run.smithi094.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-02T11:50:49.503 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 62 M 2024-09-02T11:50:49.504 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:50:49.576 INFO:teuthology.orchestra.run.smithi094.stdout:(31/32): librados2-19.3.0-4585.gb59673c4.el9.x8 16 MB/s | 3.5 MB 00:00 2024-09-02T11:50:49.751 INFO:teuthology.orchestra.run.smithi094.stdout:(32/32): librbd1-19.3.0-4585.gb59673c4.el9.x86_ 9.9 MB/s | 3.2 MB 00:00 2024-09-02T11:50:49.757 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:50:49.757 INFO:teuthology.orchestra.run.smithi094.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-02T11:50:49.862 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:50:49.898 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:50:49.899 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:50:50.262 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-base-19.3.0-4585.gb59673c4.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-02T11:50:50.329 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-selinux-19.3.0-4585.gb59673c4.el9. 378 kB/s | 25 kB 00:00 2024-09-02T11:50:50.379 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:50:50.380 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:50:50.430 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): libcephfs2-19.3.0-4585.gb59673c4.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-02T11:50:50.588 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): ceph-radosgw-19.3.0-4585.gb59673c4.el9. 13 MB/s | 11 MB 00:00 2024-09-02T11:50:50.630 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): libradosstriper1-19.3.0-4585.gb59673c4. 2.5 MB/s | 508 kB 00:00 2024-09-02T11:50:50.681 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-ceph-argparse-19.3.0-4585.gb596 902 kB/s | 45 kB 00:00 2024-09-02T11:50:50.764 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-common-19.3.0-4585.gb59673 1.7 MB/s | 143 kB 00:00 2024-09-02T11:50:50.864 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): librgw2-19.3.0-4585.gb59673c4.el9.x86_6 20 MB/s | 5.5 MB 00:00 2024-09-02T11:50:50.890 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-cephfs-19.3.0-4585.gb59673c4.el 1.3 MB/s | 162 kB 00:00 2024-09-02T11:50:50.923 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-rados-19.3.0-4585.gb59673c4.el 5.3 MB/s | 320 kB 00:00 2024-09-02T11:50:50.948 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rbd-19.3.0-4585.gb59673c4.el9. 5.1 MB/s | 305 kB 00:00 2024-09-02T11:50:50.974 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-rgw-19.3.0-4585.gb59673c4.el9. 1.9 MB/s | 99 kB 00:00 2024-09-02T11:50:51.120 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 11 M 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-09-02T11:50:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.5 M 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 22 M 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 25 k 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 755 k 2024-09-02T11:50:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 508 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 45 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 143 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 162 k 2024-09-02T11:50:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 320 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 305 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 99 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:50:51.125 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:50:51.126 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:50:51.126 INFO:teuthology.orchestra.run.smithi038.stdout:Install 30 Packages 2024-09-02T11:50:51.126 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-09-02T11:50:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:50:51.126 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 62 M 2024-09-02T11:50:51.126 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:50:51.174 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): ceph-common-19.3.0-4585.gb59673c4.el9. 15 MB/s | 22 MB 00:01 2024-09-02T11:50:51.233 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 376 kB/s | 107 kB 00:00 2024-09-02T11:50:51.258 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-09-02T11:50:51.292 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 504 kB/s | 160 kB 00:00 2024-09-02T11:50:51.310 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-02T11:50:51.333 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-02T11:50:51.367 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-02T11:50:51.392 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-02T11:50:51.418 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 714 kB/s | 36 kB 00:00 2024-09-02T11:50:51.443 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-09-02T11:50:51.468 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-02T11:50:51.527 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-02T11:50:51.552 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 373 kB/s | 26 kB 00:00 2024-09-02T11:50:51.577 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-09-02T11:50:51.602 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-02T11:50:51.636 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-02T11:50:51.670 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-02T11:50:51.729 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-02T11:50:51.854 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-02T11:50:51.882 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:50:51.961 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T11:50:51.998 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T11:50:52.023 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T11:50:52.046 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:52.067 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:52.108 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-base-19.3.0-4585.gb59673c4.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-02T11:50:52.121 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): librados2-19.3.0-4585.gb59673c4.el9.x8 7.7 MB/s | 3.5 MB 00:00 2024-09-02T11:50:52.167 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-selinux-19.3.0-4585.gb59673c4.el9. 433 kB/s | 25 kB 00:00 2024-09-02T11:50:52.179 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librbd1-19.3.0-4585.gb59673c4.el9.x86_ 7.2 MB/s | 3.2 MB 00:00 2024-09-02T11:50:52.184 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:50:52.184 INFO:teuthology.orchestra.run.smithi087.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-02T11:50:52.242 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): libcephfs2-19.3.0-4585.gb59673c4.el9.x8 9.9 MB/s | 755 kB 00:00 2024-09-02T11:50:52.281 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:50:52.309 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): libradosstriper1-19.3.0-4585.gb59673c4. 7.5 MB/s | 508 kB 00:00 2024-09-02T11:50:52.320 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:50:52.321 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:50:52.331 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:52.385 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:52.643 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): librgw2-19.3.0-4585.gb59673c4.el9.x86_6 17 MB/s | 5.5 MB 00:00 2024-09-02T11:50:52.685 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): python3-ceph-argparse-19.3.0-4585.gb596 1.1 MB/s | 45 kB 00:00 2024-09-02T11:50:52.713 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T11:50:52.795 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:50:52.795 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:50:52.841 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:52.893 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): ceph-common-19.3.0-4585.gb59673c4.el9.x 16 MB/s | 22 MB 00:01 2024-09-02T11:50:52.918 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): python3-ceph-common-19.3.0-4585.gb59673 612 kB/s | 143 kB 00:00 2024-09-02T11:50:52.933 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:52.944 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-cephfs-19.3.0-4585.gb59673c4.el 3.2 MB/s | 162 kB 00:00 2024-09-02T11:50:52.969 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-rados-19.3.0-4585.gb59673c4.el 6.3 MB/s | 320 kB 00:00 2024-09-02T11:50:52.994 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-rbd-19.3.0-4585.gb59673c4.el9. 5.9 MB/s | 305 kB 00:00 2024-09-02T11:50:53.020 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-rgw-19.3.0-4585.gb59673c4.el9. 1.9 MB/s | 99 kB 00:00 2024-09-02T11:50:53.020 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 8/34 2024-09-02T11:50:53.059 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:53.121 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:53.150 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T11:50:53.153 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): ceph-radosgw-19.3.0-4585.gb59673c4.el9 6.9 MB/s | 11 MB 00:01 2024-09-02T11:50:53.193 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 11/34 2024-09-02T11:50:53.212 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 492 kB/s | 107 kB 00:00 2024-09-02T11:50:53.245 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-09-02T11:50:53.263 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:50:53.286 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:53.296 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 581 kB/s | 160 kB 00:00 2024-09-02T11:50:53.321 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 662 kB 00:00 2024-09-02T11:50:53.346 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-02T11:50:53.350 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:53.372 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-02T11:50:53.384 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T11:50:53.397 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-02T11:50:53.439 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 710 kB/s | 47 kB 00:00 2024-09-02T11:50:53.464 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-02T11:50:53.492 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T11:50:53.523 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-09-02T11:50:53.527 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T11:50:53.548 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-02T11:50:53.567 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T11:50:53.573 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 280 kB/s | 26 kB 00:00 2024-09-02T11:50:53.592 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T11:50:53.598 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-02T11:50:53.619 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:50:53.621 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 11 M 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.5 M 2024-09-02T11:50:53.622 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 22 M 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 25 k 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T11:50:53.623 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 755 k 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 508 k 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:50:53.624 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 45 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 143 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 162 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 320 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 305 k 2024-09-02T11:50:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 99 k 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout:Installing weak dependencies: 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout:Install 30 Packages 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2024-09-02T11:50:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:50:53.627 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 62 M 2024-09-02T11:50:53.627 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:50:53.632 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-02T11:50:53.654 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T11:50:53.666 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-02T11:50:53.741 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-02T11:50:53.825 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-02T11:50:53.925 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): librados2-19.3.0-4585.gb59673c4.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-02T11:50:53.976 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T11:50:54.016 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): librbd1-19.3.0-4585.gb59673c4.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-02T11:50:54.017 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:50:54.021 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:50:54.021 INFO:teuthology.orchestra.run.smithi038.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-02T11:50:54.107 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:50:54.123 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:50:54.138 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 23/34 2024-09-02T11:50:54.162 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:50:54.162 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:50:54.174 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T11:50:54.269 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:50:54.347 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T11:50:54.385 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T11:50:54.395 INFO:teuthology.orchestra.run.smithi099.stdout:(1/32): ceph-base-19.3.0-4585.gb59673c4.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-02T11:50:54.418 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T11:50:54.445 INFO:teuthology.orchestra.run.smithi099.stdout:(2/32): ceph-selinux-19.3.0-4585.gb59673c4.el9. 504 kB/s | 25 kB 00:00 2024-09-02T11:50:54.449 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:54.472 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:54.545 INFO:teuthology.orchestra.run.smithi099.stdout:(3/32): libcephfs2-19.3.0-4585.gb59673c4.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-02T11:50:54.663 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:50:54.663 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:50:54.668 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T11:50:54.694 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:50:54.695 INFO:teuthology.orchestra.run.smithi099.stdout:(4/32): ceph-radosgw-19.3.0-4585.gb59673c4.el9. 13 MB/s | 11 MB 00:00 2024-09-02T11:50:54.729 INFO:teuthology.orchestra.run.smithi099.stdout:(5/32): libradosstriper1-19.3.0-4585.gb59673c4. 2.7 MB/s | 508 kB 00:00 2024-09-02T11:50:54.745 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:54.777 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:50:54.805 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:54.821 INFO:teuthology.orchestra.run.smithi099.stdout:(6/32): python3-ceph-argparse-19.3.0-4585.gb596 492 kB/s | 45 kB 00:00 2024-09-02T11:50:54.843 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 27/34 2024-09-02T11:50:54.876 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 28/34 2024-09-02T11:50:54.882 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.6 MB 00:01 2024-09-02T11:50:54.929 INFO:teuthology.orchestra.run.smithi099.stdout:(7/32): python3-ceph-common-19.3.0-4585.gb59673 1.3 MB/s | 143 kB 00:00 2024-09-02T11:50:55.071 INFO:teuthology.orchestra.run.smithi099.stdout:(8/32): librgw2-19.3.0-4585.gb59673c4.el9.x86_6 15 MB/s | 5.5 MB 00:00 2024-09-02T11:50:55.097 INFO:teuthology.orchestra.run.smithi099.stdout:(9/32): python3-cephfs-19.3.0-4585.gb59673c4.el 968 kB/s | 162 kB 00:00 2024-09-02T11:50:55.122 INFO:teuthology.orchestra.run.smithi099.stdout:(10/32): python3-rados-19.3.0-4585.gb59673c4.el 6.2 MB/s | 320 kB 00:00 2024-09-02T11:50:55.156 INFO:teuthology.orchestra.run.smithi099.stdout:(11/32): python3-rbd-19.3.0-4585.gb59673c4.el9. 5.1 MB/s | 305 kB 00:00 2024-09-02T11:50:55.179 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T11:50:55.181 INFO:teuthology.orchestra.run.smithi099.stdout:(12/32): python3-rgw-19.3.0-4585.gb59673c4.el9. 1.7 MB/s | 99 kB 00:00 2024-09-02T11:50:55.237 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:55.329 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:55.393 INFO:teuthology.orchestra.run.smithi099.stdout:(13/32): ceph-common-19.3.0-4585.gb59673c4.el9. 14 MB/s | 22 MB 00:01 2024-09-02T11:50:55.416 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 8/34 2024-09-02T11:50:55.440 INFO:teuthology.orchestra.run.smithi099.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 376 kB/s | 107 kB 00:00 2024-09-02T11:50:55.446 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:55.499 INFO:teuthology.orchestra.run.smithi099.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-02T11:50:55.500 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:55.524 INFO:teuthology.orchestra.run.smithi099.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 468 kB/s | 160 kB 00:00 2024-09-02T11:50:55.529 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T11:50:55.572 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 11/34 2024-09-02T11:50:55.584 INFO:teuthology.orchestra.run.smithi099.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-02T11:50:55.624 INFO:teuthology.orchestra.run.smithi099.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-02T11:50:55.642 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:50:55.650 INFO:teuthology.orchestra.run.smithi099.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-02T11:50:55.665 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:55.675 INFO:teuthology.orchestra.run.smithi099.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-02T11:50:55.700 INFO:teuthology.orchestra.run.smithi099.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-09-02T11:50:55.729 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:55.751 INFO:teuthology.orchestra.run.smithi099.stdout:(22/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-02T11:50:55.763 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T11:50:55.793 INFO:teuthology.orchestra.run.smithi099.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-02T11:50:55.818 INFO:teuthology.orchestra.run.smithi099.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-02T11:50:55.843 INFO:teuthology.orchestra.run.smithi099.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 293 kB/s | 26 kB 00:00 2024-09-02T11:50:55.868 INFO:teuthology.orchestra.run.smithi099.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-02T11:50:55.870 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T11:50:55.902 INFO:teuthology.orchestra.run.smithi099.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-02T11:50:55.906 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T11:50:55.952 INFO:teuthology.orchestra.run.smithi099.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-02T11:50:55.953 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T11:50:55.988 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T11:50:56.004 INFO:teuthology.orchestra.run.smithi099.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-02T11:50:56.017 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T11:50:56.049 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T11:50:56.095 INFO:teuthology.orchestra.run.smithi099.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-02T11:50:56.162 INFO:teuthology.orchestra.run.smithi099.stdout:(31/32): librados2-19.3.0-4585.gb59673c4.el9.x8 17 MB/s | 3.5 MB 00:00 2024-09-02T11:50:56.162 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:50:56.243 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T11:50:56.262 INFO:teuthology.orchestra.run.smithi099.stdout:(32/32): librbd1-19.3.0-4585.gb59673c4.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-02T11:50:56.267 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:50:56.267 INFO:teuthology.orchestra.run.smithi099.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-02T11:50:56.280 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T11:50:56.305 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T11:50:56.328 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:56.355 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T11:50:56.370 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:50:56.406 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:50:56.406 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:50:56.413 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:50:56.504 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:50:56.534 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 23/34 2024-09-02T11:50:56.566 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 18 kB 00:00 2024-09-02T11:50:56.570 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T11:50:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:56.692 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:56.880 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:50:56.881 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:50:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T11:50:57.059 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T11:50:57.095 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:50:57.165 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:50:57.166 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:57.203 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:50:57.235 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 27/34 2024-09-02T11:50:57.236 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:50:57.258 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:57.263 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 28/34 2024-09-02T11:50:57.349 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 8/34 2024-09-02T11:50:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T11:50:57.495 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-02T11:50:57.512 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 11/34 2024-09-02T11:50:57.582 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:50:57.620 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:57.684 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:57.718 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T11:50:57.785 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:50:57.829 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T11:50:57.840 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:50:57.860 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T11:50:57.901 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T11:50:57.925 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:50:57.926 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T11:50:57.926 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T11:50:57.926 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:50:57.926 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T11:50:57.955 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T11:50:57.957 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:50:57.987 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T11:50:58.323 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T11:50:58.337 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:50:58.367 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:50:58.429 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T11:50:58.458 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:50:58.466 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T11:50:58.488 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 23/34 2024-09-02T11:50:58.492 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T11:50:58.515 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:58.525 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T11:50:58.538 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:50:58.808 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:58.862 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:50:59.051 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T11:50:59.195 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:50:59.250 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T11:50:59.266 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:50:59.311 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 27/34 2024-09-02T11:50:59.380 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 28/34 2024-09-02T11:50:59.410 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:50:59.506 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 8/34 2024-09-02T11:50:59.537 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:59.591 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:50:59.604 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:50:59.620 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T11:50:59.664 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 11/34 2024-09-02T11:50:59.732 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:50:59.756 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:59.768 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:50:59.804 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:50:59.838 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T11:50:59.947 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T11:50:59.980 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T11:51:00.020 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T11:51:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T11:51:00.075 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T11:51:00.116 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T11:51:00.323 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:00.371 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:00.436 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T11:51:00.465 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:00.465 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T11:51:00.465 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T11:51:00.465 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:00.479 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:51:00.497 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:00.577 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:51:00.609 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 23/34 2024-09-02T11:51:00.645 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T11:51:01.141 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T11:51:01.170 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:51:01.241 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:51:01.329 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 27/34 2024-09-02T11:51:01.387 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 28/34 2024-09-02T11:51:01.788 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:01.826 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:02.413 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:02.446 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:02.539 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:02.539 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T11:51:02.539 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T11:51:02.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:02.573 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:03.084 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-02T11:51:03.610 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:03.683 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:04.232 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:04.320 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:04.412 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:04.412 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T11:51:04.412 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T11:51:04.412 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:04.446 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:05.514 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 11 M 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.5 M 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:51:05.517 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 22 M 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 25 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 755 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T11:51:05.518 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 508 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 5.5 M 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 45 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 143 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 162 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 320 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 305 k 2024-09-02T11:51:05.519 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 99 k 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-09-02T11:51:05.520 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:05.521 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 62 M 2024-09-02T11:51:05.521 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:51:06.369 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-base-19.3.0-4585.gb59673c4.el9.x86 9.3 MB/s | 5.5 MB 00:00 2024-09-02T11:51:06.436 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.3.0-4585.gb59673c4.el9. 379 kB/s | 25 kB 00:00 2024-09-02T11:51:06.570 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): ceph-radosgw-19.3.0-4585.gb59673c4.el9. 14 MB/s | 11 MB 00:00 2024-09-02T11:51:06.603 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libcephfs2-19.3.0-4585.gb59673c4.el9.x8 4.4 MB/s | 755 kB 00:00 2024-09-02T11:51:06.637 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): libradosstriper1-19.3.0-4585.gb59673c4. 7.5 MB/s | 508 kB 00:00 2024-09-02T11:51:06.695 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.3.0-4585.gb596 774 kB/s | 45 kB 00:00 2024-09-02T11:51:06.754 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.3.0-4585.gb59673 2.4 MB/s | 143 kB 00:00 2024-09-02T11:51:06.812 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.3.0-4585.gb59673c4.el 2.7 MB/s | 162 kB 00:00 2024-09-02T11:51:06.896 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): librgw2-19.3.0-4585.gb59673c4.el9.x86_6 19 MB/s | 5.5 MB 00:00 2024-09-02T11:51:06.921 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rados-19.3.0-4585.gb59673c4.el 2.9 MB/s | 320 kB 00:00 2024-09-02T11:51:06.946 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rbd-19.3.0-4585.gb59673c4.el9. 5.9 MB/s | 305 kB 00:00 2024-09-02T11:51:06.972 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): python3-rgw-19.3.0-4585.gb59673c4.el9. 1.9 MB/s | 99 kB 00:00 2024-09-02T11:51:07.189 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): ceph-common-19.3.0-4585.gb59673c4.el9. 15 MB/s | 22 MB 00:01 2024-09-02T11:51:07.247 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-09-02T11:51:07.281 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-09-02T11:51:07.331 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-02T11:51:07.365 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-02T11:51:07.390 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-09-02T11:51:07.416 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-02T11:51:07.441 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 715 kB/s | 36 kB 00:00 2024-09-02T11:51:07.466 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-02T11:51:07.491 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-09-02T11:51:07.550 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-02T11:51:07.575 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-09-02T11:51:07.600 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-09-02T11:51:07.625 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-02T11:51:07.684 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-02T11:51:07.709 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-02T11:51:07.785 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-02T11:51:07.868 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-02T11:51:08.027 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librados2-19.3.0-4585.gb59673c4.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /root 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2024-09-02T11:51:08.062 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:08.144 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): librbd1-19.3.0-4585.gb59673c4.el9.x86_ 9.0 MB/s | 3.2 MB 00:00 2024-09-02T11:51:08.149 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:08.149 INFO:teuthology.orchestra.run.smithi064.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-02T11:51:08.254 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:51:08.294 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:51:08.294 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:51:08.793 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:51:08.793 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:51:09.093 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:09.118 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:09.118 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T11:51:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:09.119 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:09.119 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:09.120 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:09.228 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:10.307 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:51:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T11:51:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T11:51:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T11:51:10.460 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-09-02T11:51:10.461 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:51:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 1/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 3/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 4/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 5/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 6/34 2024-09-02T11:51:10.555 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 8/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 9/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 10/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 11/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 13/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T11:51:10.556 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:10.557 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T11:51:10.558 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 33/34 2024-09-02T11:51:10.746 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:51:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T11:51:11.213 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T11:51:11.267 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.302 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T11:51:11.303 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:11.304 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 8/34 2024-09-02T11:51:11.470 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:51:11.495 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:11.495 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:11.496 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T11:51:11.496 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:11.496 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:11.496 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:11.497 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:11.540 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 9/34 2024-09-02T11:51:11.569 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T11:51:11.581 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:11.581 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:11.589 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-test 2024-09-02T11:51:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 11/34 2024-09-02T11:51:11.683 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:51:11.713 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:51:11.777 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 13/34 2024-09-02T11:51:11.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T11:51:11.922 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T11:51:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T11:51:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T11:51:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T11:51:12.082 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T11:51:12.123 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T11:51:12.166 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:28 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:12.313 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:12.314 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:12.314 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:51:12.314 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:12.314 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:51:12.314 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 50 M 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 161 k 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout:Install 4 Packages 2024-09-02T11:51:12.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:12.316 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 51 M 2024-09-02T11:51:12.316 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 216 M 2024-09-02T11:51:12.316 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:51:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T11:51:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:51:12.594 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 22/34 2024-09-02T11:51:12.621 INFO:teuthology.orchestra.run.smithi094.stdout:(1/4): libcephsqlite-19.3.0-4585.gb59673c4.el9. 789 kB/s | 161 kB 00:00 2024-09-02T11:51:12.623 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 23/34 2024-09-02T11:51:12.660 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T11:51:12.697 INFO:teuthology.orchestra.run.smithi094.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-02T11:51:12.831 INFO:teuthology.orchestra.run.smithi094.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 304 kB/s | 64 kB 00:00 2024-09-02T11:51:12.845 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:12.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 1/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 3/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 4/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 5/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 6/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 8/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 9/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 10/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 11/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:51:12.846 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 13/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T11:51:12.847 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T11:51:12.848 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 33/34 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-09-02T11:51:13.092 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-09-02T11:51:13.093 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-09-02T11:51:13.093 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T11:51:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:51:13.275 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 26/34 2024-09-02T11:51:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 27/34 2024-09-02T11:51:13.398 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 28/34 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.647 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T11:51:13.648 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T11:51:13.649 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T11:51:13.650 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T11:51:13.650 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T11:51:13.650 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T11:51:13.650 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:13.650 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:13.912 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-09-02T11:51:13.999 INFO:teuthology.orchestra.run.smithi094.stdout:(4/4): ceph-test-19.3.0-4585.gb59673c4.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-02T11:51:14.001 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:14.002 INFO:teuthology.orchestra.run.smithi094.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-02T11:51:14.029 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:51:14.049 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:51:14.049 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:51:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:14.172 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:14.172 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T11:51:14.172 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:14.173 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:14.257 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:14.258 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:14.432 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:51:14.432 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:51:14.543 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:27 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:14.640 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:14.640 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /root 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2024-09-02T11:51:14.641 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:14.681 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:14.681 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 50 M 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 161 k 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:14.682 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-09-02T11:51:14.683 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:14.683 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 51 M 2024-09-02T11:51:14.683 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 216 M 2024-09-02T11:51:14.683 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:51:14.863 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-02T11:51:14.930 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 950 kB/s | 64 kB 00:00 2024-09-02T11:51:14.962 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:51:14.980 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): libcephsqlite-19.3.0-4585.gb59673c4.el9. 585 kB/s | 161 kB 00:00 2024-09-02T11:51:15.035 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T11:51:15.074 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T11:51:15.103 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:15.499 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:15.499 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 1/34 2024-09-02T11:51:15.499 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2/34 2024-09-02T11:51:15.499 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 3/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 4/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 5/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 6/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 8/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 9/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 10/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 11/34 2024-09-02T11:51:15.500 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 13/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T11:51:15.501 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T11:51:15.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T11:51:15.503 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:15.503 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T11:51:15.503 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 33/34 2024-09-02T11:51:15.669 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:15.696 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:15.696 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:15.696 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T11:51:15.696 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:15.697 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:15.697 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:15.698 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:15.780 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:15.780 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T11:51:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T11:51:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T11:51:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T11:51:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:16.328 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:16.382 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): ceph-test-19.3.0-4585.gb59673c4.el9.x86_ 30 MB/s | 50 MB 00:01 2024-09-02T11:51:16.384 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:16.384 INFO:teuthology.orchestra.run.smithi087.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-02T11:51:16.428 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:51:16.448 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:51:16.448 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:51:16.464 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 29/34 2024-09-02T11:51:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:16.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-09-02T11:51:16.593 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 30/34 2024-09-02T11:51:16.593 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T11:51:16.593 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T11:51:16.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:16.624 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:16.842 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:51:16.842 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 1/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 3/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 4/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 5/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 6/34 2024-09-02T11:51:17.038 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 8/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 9/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 10/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 11/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 13/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T11:51:17.039 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:17.040 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T11:51:17.041 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 33/34 2024-09-02T11:51:17.187 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:29 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:17.330 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:17.331 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:17.331 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:51:17.331 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:17.331 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:51:17.331 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 50 M 2024-09-02T11:51:17.331 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 161 k 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-09-02T11:51:17.332 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:17.333 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 51 M 2024-09-02T11:51:17.333 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 216 M 2024-09-02T11:51:17.333 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:51:17.400 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:51:17.468 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T11:51:17.516 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T11:51:17.545 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:17.599 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): libcephsqlite-19.3.0-4585.gb59673c4.el9. 752 kB/s | 161 kB 00:00 2024-09-02T11:51:17.657 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-02T11:51:17.809 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 303 kB/s | 64 kB 00:00 2024-09-02T11:51:17.854 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:17.854 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:17.854 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2024-09-02T11:51:17.854 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.854 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T11:51:17.855 INFO:teuthology.orchestra.run.smithi099.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.856 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:17.857 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:18.225 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2024-09-02T11:51:18.798 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:27 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:18.954 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:18.954 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 50 M 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 161 k 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:18.955 INFO:teuthology.orchestra.run.smithi099.stdout:Install 4 Packages 2024-09-02T11:51:18.956 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:18.956 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 51 M 2024-09-02T11:51:18.956 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 216 M 2024-09-02T11:51:18.956 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:51:19.060 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): ceph-test-19.3.0-4585.gb59673c4.el9.x86_ 30 MB/s | 50 MB 00:01 2024-09-02T11:51:19.062 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:19.062 INFO:teuthology.orchestra.run.smithi038.stdout:Total 29 MB/s | 51 MB 00:01 2024-09-02T11:51:19.090 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:51:19.112 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:51:19.112 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:51:19.431 INFO:teuthology.orchestra.run.smithi099.stdout:(1/4): libcephsqlite-19.3.0-4585.gb59673c4.el9. 762 kB/s | 161 kB 00:00 2024-09-02T11:51:19.507 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:51:19.508 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:51:19.826 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:19.899 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:19.995 INFO:teuthology.orchestra.run.smithi099.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 392 kB/s | 304 kB 00:00 2024-09-02T11:51:20.052 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:51:20.079 INFO:teuthology.orchestra.run.smithi099.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 98 kB/s | 64 kB 00:00 2024-09-02T11:51:20.131 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T11:51:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T11:51:20.197 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:20.765 INFO:teuthology.orchestra.run.smithi099.stdout:(4/4): ceph-test-19.3.0-4585.gb59673c4.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-02T11:51:20.768 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:20.768 INFO:teuthology.orchestra.run.smithi099.stdout:Total 28 MB/s | 51 MB 00:01 2024-09-02T11:51:20.795 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:51:20.816 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:51:20.816 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:51:21.194 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:51:21.194 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:51:21.730 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:51:21.807 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T11:51:21.856 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T11:51:21.884 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:22.232 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:22.307 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:22.489 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:22.489 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 1/4 2024-09-02T11:51:22.489 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2/4 2024-09-02T11:51:22.489 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T11:51:22.755 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:22.756 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:23.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph 2024-09-02T11:51:23.630 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:39 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:23.771 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 6.6 k 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 2.4 M 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 1.5 M 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 291 k 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 4.8 M 2024-09-02T11:51:23.774 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 17 M 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T11:51:23.775 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T11:51:23.776 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T11:51:23.777 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:23.778 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:51:23.778 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:23.778 INFO:teuthology.orchestra.run.smithi094.stdout:Install 37 Packages 2024-09-02T11:51:23.778 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:23.778 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 31 M 2024-09-02T11:51:23.778 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 109 M 2024-09-02T11:51:23.779 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:51:24.332 INFO:teuthology.orchestra.run.smithi094.stdout:(1/37): ceph-19.3.0-4585.gb59673c4.el9.x86_64.r 56 kB/s | 6.6 kB 00:00 2024-09-02T11:51:24.507 INFO:teuthology.orchestra.run.smithi094.stdout:(2/37): ceph-mgr-19.3.0-4585.gb59673c4.el9.x86_ 5.2 MB/s | 1.5 MB 00:00 2024-09-02T11:51:24.566 INFO:teuthology.orchestra.run.smithi094.stdout:(3/37): ceph-mds-19.3.0-4585.gb59673c4.el9.x86_ 6.7 MB/s | 2.4 MB 00:00 2024-09-02T11:51:24.641 INFO:teuthology.orchestra.run.smithi094.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4585.gb596 3.8 MB/s | 291 kB 00:00 2024-09-02T11:51:24.725 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:24.725 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 1/4 2024-09-02T11:51:24.725 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2/4 2024-09-02T11:51:24.725 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T11:51:24.741 INFO:teuthology.orchestra.run.smithi094.stdout:(5/37): ceph-mon-19.3.0-4585.gb59673c4.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:25.014 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:25.083 INFO:teuthology.orchestra.run.smithi094.stdout:(6/37): ceph-osd-19.3.0-4585.gb59673c4.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-02T11:51:25.149 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:25.230 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:25.326 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-09-02T11:51:25.342 INFO:teuthology.orchestra.run.smithi094.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 361 kB/s | 253 kB 00:00 2024-09-02T11:51:25.501 INFO:teuthology.orchestra.run.smithi094.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 255 kB/s | 106 kB 00:00 2024-09-02T11:51:25.534 INFO:teuthology.orchestra.run.smithi094.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.5 MB/s | 1.2 MB 00:00 2024-09-02T11:51:25.760 INFO:teuthology.orchestra.run.smithi094.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 324 kB/s | 135 kB 00:00 2024-09-02T11:51:25.879 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:38 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:25.926 INFO:teuthology.orchestra.run.smithi094.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 293 kB/s | 125 kB 00:00 2024-09-02T11:51:26.016 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:26.018 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:26.018 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:51:26.018 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:26.018 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 6.6 k 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 2.4 M 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 1.5 M 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 291 k 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 4.8 M 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 17 M 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T11:51:26.019 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T11:51:26.020 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T11:51:26.021 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout:Install 37 Packages 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:26.022 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 M 2024-09-02T11:51:26.023 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 109 M 2024-09-02T11:51:26.023 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:51:26.027 INFO:teuthology.orchestra.run.smithi094.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 436 kB/s | 215 kB 00:00 2024-09-02T11:51:26.590 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:26.663 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:26.702 INFO:teuthology.orchestra.run.smithi094.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 200 kB/s | 188 kB 00:00 2024-09-02T11:51:26.728 INFO:teuthology.orchestra.run.smithi094.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 50 kB/s | 35 kB 00:00 2024-09-02T11:51:26.961 INFO:teuthology.orchestra.run.smithi094.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 166 kB/s | 172 kB 00:01 2024-09-02T11:51:27.050 INFO:teuthology.orchestra.run.smithi087.stdout:(1/37): ceph-19.3.0-4585.gb59673c4.el9.x86_64.r 53 kB/s | 6.6 kB 00:00 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-09-02T11:51:27.094 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:27.250 INFO:teuthology.orchestra.run.smithi087.stdout:(2/37): ceph-mgr-19.3.0-4585.gb59673c4.el9.x86_ 4.7 MB/s | 1.5 MB 00:00 2024-09-02T11:51:27.317 INFO:teuthology.orchestra.run.smithi087.stdout:(3/37): ceph-mds-19.3.0-4585.gb59673c4.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-09-02T11:51:27.361 INFO:teuthology.orchestra.run.smithi094.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-09-02T11:51:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 113 kB/s | 77 kB 00:00 2024-09-02T11:51:27.429 INFO:teuthology.orchestra.run.smithi094.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-09-02T11:51:27.451 INFO:teuthology.orchestra.run.smithi087.stdout:(4/37): ceph-mon-19.3.0-4585.gb59673c4.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-02T11:51:27.453 INFO:teuthology.orchestra.run.smithi094.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-09-02T11:51:27.476 INFO:teuthology.orchestra.run.smithi087.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4585.gb596 1.8 MB/s | 291 kB 00:00 2024-09-02T11:51:27.479 INFO:teuthology.orchestra.run.smithi094.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 868 kB/s | 43 kB 00:00 2024-09-02T11:51:27.521 INFO:teuthology.orchestra.run.smithi094.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-02T11:51:27.546 INFO:teuthology.orchestra.run.smithi094.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 463 kB/s | 11 kB 00:00 2024-09-02T11:51:27.571 INFO:teuthology.orchestra.run.smithi094.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 706 kB/s | 18 kB 00:00 2024-09-02T11:51:27.597 INFO:teuthology.orchestra.run.smithi094.stdout:(24/37): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-09-02T11:51:27.610 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:27.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 1/4 2024-09-02T11:51:27.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2/4 2024-09-02T11:51:27.610 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T11:51:27.622 INFO:teuthology.orchestra.run.smithi094.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 471 kB/s | 23 kB 00:00 2024-09-02T11:51:27.626 INFO:teuthology.orchestra.run.smithi087.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-02T11:51:27.656 INFO:teuthology.orchestra.run.smithi094.stdout:(26/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 851 kB/s | 172 kB 00:00 2024-09-02T11:51:27.677 INFO:teuthology.orchestra.run.smithi087.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-02T11:51:27.681 INFO:teuthology.orchestra.run.smithi094.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 384 kB/s | 20 kB 00:00 2024-09-02T11:51:27.715 INFO:teuthology.orchestra.run.smithi094.stdout:(28/37): python3-jaraco-functools-3.5.0-2.el9.n 165 kB/s | 19 kB 00:00 2024-09-02T11:51:27.740 INFO:teuthology.orchestra.run.smithi094.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 565 kB/s | 46 kB 00:00 2024-09-02T11:51:27.765 INFO:teuthology.orchestra.run.smithi094.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 661 kB/s | 16 kB 00:00 2024-09-02T11:51:27.790 INFO:teuthology.orchestra.run.smithi094.stdout:(31/37): python3-more-itertools-8.12.0-2.el9.no 722 kB/s | 79 kB 00:00 2024-09-02T11:51:27.824 INFO:teuthology.orchestra.run.smithi094.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-02T11:51:27.849 INFO:teuthology.orchestra.run.smithi094.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-09-02T11:51:27.852 INFO:teuthology.orchestra.run.smithi087.stdout:(8/37): ceph-osd-19.3.0-4585.gb59673c4.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-02T11:51:27.874 INFO:teuthology.orchestra.run.smithi094.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-02T11:51:27.877 INFO:teuthology.orchestra.run.smithi087.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-02T11:51:27.900 INFO:teuthology.orchestra.run.smithi094.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-09-02T11:51:27.911 INFO:teuthology.orchestra.run.smithi087.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-02T11:51:27.925 INFO:teuthology.orchestra.run.smithi094.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-02T11:51:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:27.926 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:27.936 INFO:teuthology.orchestra.run.smithi087.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-02T11:51:27.959 INFO:teuthology.orchestra.run.smithi094.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-02T11:51:27.961 INFO:teuthology.orchestra.run.smithi087.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-09-02T11:51:27.965 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:27.966 INFO:teuthology.orchestra.run.smithi094.stdout:Total 7.3 MB/s | 31 MB 00:04 2024-09-02T11:51:27.995 INFO:teuthology.orchestra.run.smithi087.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-02T11:51:28.020 INFO:teuthology.orchestra.run.smithi087.stdout:(14/37): python3-requests-2.25.1-8.el9.noarch.r 744 kB/s | 125 kB 00:00 2024-09-02T11:51:28.045 INFO:teuthology.orchestra.run.smithi087.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 414 kB/s | 35 kB 00:00 2024-09-02T11:51:28.071 INFO:teuthology.orchestra.run.smithi087.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-02T11:51:28.096 INFO:teuthology.orchestra.run.smithi087.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-09-02T11:51:28.119 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:51:28.121 INFO:teuthology.orchestra.run.smithi087.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-02T11:51:28.140 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:28.147 INFO:teuthology.orchestra.run.smithi087.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-09-02T11:51:28.159 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:51:28.159 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:51:28.165 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 32/34 2024-09-02T11:51:28.166 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:28.166 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T11:51:28.166 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:28.166 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T11:51:28.166 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:28.168 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:28.197 INFO:teuthology.orchestra.run.smithi087.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-02T11:51:28.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-09-02T11:51:28.222 INFO:teuthology.orchestra.run.smithi087.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-02T11:51:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T11:51:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:28.247 INFO:teuthology.orchestra.run.smithi087.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-02T11:51:28.287 INFO:teuthology.orchestra.run.smithi087.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 456 kB/s | 18 kB 00:00 2024-09-02T11:51:28.323 INFO:teuthology.orchestra.run.smithi087.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-02T11:51:28.348 INFO:teuthology.orchestra.run.smithi087.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 379 kB/s | 23 kB 00:00 2024-09-02T11:51:28.374 INFO:teuthology.orchestra.run.smithi087.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-02T11:51:28.399 INFO:teuthology.orchestra.run.smithi087.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-02T11:51:28.424 INFO:teuthology.orchestra.run.smithi087.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-02T11:51:28.457 INFO:teuthology.orchestra.run.smithi087.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-02T11:51:28.459 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:51:28.460 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:51:28.483 INFO:teuthology.orchestra.run.smithi087.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 942 kB/s | 79 kB 00:00 2024-09-02T11:51:28.508 INFO:teuthology.orchestra.run.smithi087.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-02T11:51:28.533 INFO:teuthology.orchestra.run.smithi087.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-02T11:51:28.559 INFO:teuthology.orchestra.run.smithi087.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-02T11:51:28.584 INFO:teuthology.orchestra.run.smithi087.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-02T11:51:28.609 INFO:teuthology.orchestra.run.smithi087.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-02T11:51:28.634 INFO:teuthology.orchestra.run.smithi087.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-02T11:51:28.660 INFO:teuthology.orchestra.run.smithi087.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-02T11:51:28.665 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:28.665 INFO:teuthology.orchestra.run.smithi087.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-02T11:51:28.787 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:40 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:28.817 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:51:28.858 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:51:28.858 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:51:28.928 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 6.6 k 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 2.4 M 2024-09-02T11:51:28.931 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 1.5 M 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 291 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 4.8 M 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 17 M 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T11:51:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T11:51:28.933 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T11:51:28.934 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout:Install 37 Packages 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 31 M 2024-09-02T11:51:28.935 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 109 M 2024-09-02T11:51:28.936 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:51:29.156 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:51:29.156 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:51:29.205 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:29.206 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 1/4 2024-09-02T11:51:29.206 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2/4 2024-09-02T11:51:29.206 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T11:51:29.379 INFO:teuthology.orchestra.run.smithi038.stdout:(1/37): ceph-19.3.0-4585.gb59673c4.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-02T11:51:29.447 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:29.447 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 1/34 2024-09-02T11:51:29.447 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2/34 2024-09-02T11:51:29.447 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 3/34 2024-09-02T11:51:29.447 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 4/34 2024-09-02T11:51:29.447 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 5/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_6 6/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 7/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9. 8/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x8 9/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 10/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 11/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 12/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 13/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T11:51:29.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T11:51:29.449 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T11:51:29.450 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T11:51:29.450 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 31/34 2024-09-02T11:51:29.450 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T11:51:29.450 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 33/34 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T11:51:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T11:51:29.513 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:29.513 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:29.555 INFO:teuthology.orchestra.run.smithi038.stdout:(2/37): ceph-mgr-19.3.0-4585.gb59673c4.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-02T11:51:29.631 INFO:teuthology.orchestra.run.smithi038.stdout:(3/37): ceph-mds-19.3.0-4585.gb59673c4.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-09-02T11:51:29.714 INFO:teuthology.orchestra.run.smithi038.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4585.gb596 3.4 MB/s | 291 kB 00:00 2024-09-02T11:51:29.798 INFO:teuthology.orchestra.run.smithi038.stdout:(5/37): ceph-mon-19.3.0-4585.gb59673c4.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-02T11:51:29.802 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2024-09-02T11:51:29.805 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:51:29.870 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T11:51:29.903 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T11:51:29.931 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T11:51:29.973 INFO:teuthology.orchestra.run.smithi038.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 979 kB/s | 253 kB 00:00 2024-09-02T11:51:30.004 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T11:51:30.047 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T11:51:30.080 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T11:51:30.172 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T11:51:30.173 INFO:teuthology.orchestra.run.smithi038.stdout:(7/37): ceph-osd-19.3.0-4585.gb59673c4.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-02T11:51:30.198 INFO:teuthology.orchestra.run.smithi038.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 473 kB/s | 106 kB 00:00 2024-09-02T11:51:30.215 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.216 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T11:51:30.217 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T11:51:30.218 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T11:51:30.219 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:30.219 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:51:30.250 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T11:51:30.257 INFO:teuthology.orchestra.run.smithi038.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-02T11:51:30.291 INFO:teuthology.orchestra.run.smithi038.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.5 MB/s | 1.2 MB 00:00 2024-09-02T11:51:30.300 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T11:51:30.316 INFO:teuthology.orchestra.run.smithi038.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-02T11:51:30.334 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T11:51:30.357 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:39 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:30.365 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T11:51:30.375 INFO:teuthology.orchestra.run.smithi038.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 673 kB/s | 135 kB 00:00 2024-09-02T11:51:30.407 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T11:51:30.440 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T11:51:30.481 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T11:51:30.483 INFO:teuthology.orchestra.run.smithi038.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-02T11:51:30.490 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:30.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 6.6 k 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 2.4 M 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 1.5 M 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 291 k 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 4.8 M 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 17 M 2024-09-02T11:51:30.493 INFO:teuthology.orchestra.run.smithi099.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-collections 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T11:51:30.494 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T11:51:30.495 INFO:teuthology.orchestra.run.smithi099.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout:Install 37 Packages 2024-09-02T11:51:30.496 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:30.497 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 31 M 2024-09-02T11:51:30.498 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 109 M 2024-09-02T11:51:30.498 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:51:30.504 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:51:30.516 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T11:51:30.517 INFO:teuthology.orchestra.run.smithi038.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2024-09-02T11:51:30.543 INFO:teuthology.orchestra.run.smithi038.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 750 kB/s | 188 kB 00:00 2024-09-02T11:51:30.550 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T11:51:30.567 INFO:teuthology.orchestra.run.smithi038.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 920 kB/s | 77 kB 00:00 2024-09-02T11:51:30.571 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T11:51:30.593 INFO:teuthology.orchestra.run.smithi038.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-09-02T11:51:30.599 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T11:51:30.603 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T11:51:30.618 INFO:teuthology.orchestra.run.smithi038.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-09-02T11:51:30.630 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T11:51:30.631 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T11:51:30.668 INFO:teuthology.orchestra.run.smithi038.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-09-02T11:51:30.694 INFO:teuthology.orchestra.run.smithi038.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-09-02T11:51:30.703 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T11:51:30.704 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T11:51:30.727 INFO:teuthology.orchestra.run.smithi038.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-02T11:51:30.747 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T11:51:30.752 INFO:teuthology.orchestra.run.smithi038.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 466 kB/s | 11 kB 00:00 2024-09-02T11:51:30.777 INFO:teuthology.orchestra.run.smithi038.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 715 kB/s | 18 kB 00:00 2024-09-02T11:51:30.780 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T11:51:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-02T11:51:30.828 INFO:teuthology.orchestra.run.smithi038.stdout:(25/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-02T11:51:30.852 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T11:51:30.853 INFO:teuthology.orchestra.run.smithi038.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-09-02T11:51:30.872 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T11:51:30.878 INFO:teuthology.orchestra.run.smithi038.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-09-02T11:51:30.898 INFO:teuthology.orchestra.run.smithi099.stdout:(1/37): ceph-19.3.0-4585.gb59673c4.el9.x86_64.r 55 kB/s | 6.6 kB 00:00 2024-09-02T11:51:30.904 INFO:teuthology.orchestra.run.smithi038.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-02T11:51:30.929 INFO:teuthology.orchestra.run.smithi038.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-09-02T11:51:30.939 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T11:51:30.954 INFO:teuthology.orchestra.run.smithi038.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-02T11:51:30.958 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T11:51:30.980 INFO:teuthology.orchestra.run.smithi038.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-02T11:51:30.993 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T11:51:30.999 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T11:51:31.005 INFO:teuthology.orchestra.run.smithi038.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-02T11:51:31.030 INFO:teuthology.orchestra.run.smithi038.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-09-02T11:51:31.031 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T11:51:31.034 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T11:51:31.055 INFO:teuthology.orchestra.run.smithi038.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-02T11:51:31.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T11:51:31.070 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:28 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:51:31.080 INFO:teuthology.orchestra.run.smithi038.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-09-02T11:51:31.088 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T11:51:31.098 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T11:51:31.105 INFO:teuthology.orchestra.run.smithi038.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-02T11:51:31.107 INFO:teuthology.orchestra.run.smithi099.stdout:(2/37): ceph-mgr-19.3.0-4585.gb59673c4.el9.x86_ 4.6 MB/s | 1.5 MB 00:00 2024-09-02T11:51:31.131 INFO:teuthology.orchestra.run.smithi038.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-02T11:51:31.135 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:31.135 INFO:teuthology.orchestra.run.smithi038.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-02T11:51:31.140 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T11:51:31.152 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T11:51:31.165 INFO:teuthology.orchestra.run.smithi099.stdout:(3/37): ceph-mds-19.3.0-4585.gb59673c4.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-09-02T11:51:31.180 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T11:51:31.212 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:51:31.212 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:31.212 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 50 M 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 161 k 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:51:31.213 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:31.214 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-02T11:51:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:31.214 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 51 M 2024-09-02T11:51:31.214 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 216 M 2024-09-02T11:51:31.214 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:51:31.215 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T11:51:31.241 INFO:teuthology.orchestra.run.smithi099.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4585.gb596 3.8 MB/s | 291 kB 00:00 2024-09-02T11:51:31.250 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T11:51:31.286 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:51:31.299 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T11:51:31.328 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:51:31.328 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:51:31.330 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T11:51:31.341 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T11:51:31.375 INFO:teuthology.orchestra.run.smithi099.stdout:(5/37): ceph-mon-19.3.0-4585.gb59673c4.el9.x86_ 10 MB/s | 4.8 MB 00:00 2024-09-02T11:51:31.393 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T11:51:31.403 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T11:51:31.471 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T11:51:31.543 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T11:51:31.559 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T11:51:31.600 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T11:51:31.630 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T11:51:31.642 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:51:31.643 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:51:31.682 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T11:51:31.696 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-02T11:51:31.700 INFO:teuthology.orchestra.run.smithi099.stdout:(6/37): ceph-osd-19.3.0-4585.gb59673c4.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-02T11:51:31.721 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T11:51:31.738 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): libcephsqlite-19.3.0-4585.gb59673c4.el9. 602 kB/s | 161 kB 00:00 2024-09-02T11:51:31.739 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T11:51:31.770 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T11:51:31.781 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 757 kB/s | 64 kB 00:00 2024-09-02T11:51:31.826 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T11:51:31.898 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 32/37 2024-09-02T11:51:31.928 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:31.953 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:31.954 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:31.954 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T11:51:31.954 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:31.954 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:31.954 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:32.012 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T11:51:32.084 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T11:51:32.162 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T11:51:32.251 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T11:51:32.291 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T11:51:32.300 INFO:teuthology.orchestra.run.smithi099.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 239 kB/s | 253 kB 00:01 2024-09-02T11:51:32.432 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T11:51:32.442 INFO:teuthology.orchestra.run.smithi099.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-09-02T11:51:32.492 INFO:teuthology.orchestra.run.smithi099.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-02T11:51:32.604 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 32/37 2024-09-02T11:51:32.635 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:32.661 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:32.661 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:32.662 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T11:51:32.662 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:32.662 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:32.662 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:32.992 INFO:teuthology.orchestra.run.smithi099.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 195 kB/s | 135 kB 00:00 2024-09-02T11:51:33.045 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:51:33.092 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T11:51:33.107 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-19.3.0-4585.gb59673c4.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-02T11:51:33.109 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:33.109 INFO:teuthology.orchestra.run.smithi064.stdout:Total 27 MB/s | 51 MB 00:01 2024-09-02T11:51:33.126 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T11:51:33.137 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:51:33.142 INFO:teuthology.orchestra.run.smithi099.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 178 kB/s | 125 kB 00:00 2024-09-02T11:51:33.159 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:51:33.160 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:51:33.161 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T11:51:33.239 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T11:51:33.243 INFO:teuthology.orchestra.run.smithi099.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 754 kB/s | 188 kB 00:00 2024-09-02T11:51:33.286 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T11:51:33.301 INFO:teuthology.orchestra.run.smithi099.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 597 kB/s | 35 kB 00:00 2024-09-02T11:51:33.319 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T11:51:33.326 INFO:teuthology.orchestra.run.smithi099.stdout:(14/37): python3-urllib3-1.26.5-6.el9.noarch.rp 257 kB/s | 215 kB 00:00 2024-09-02T11:51:33.360 INFO:teuthology.orchestra.run.smithi099.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 793 kB/s | 172 kB 00:00 2024-09-02T11:51:33.385 INFO:teuthology.orchestra.run.smithi099.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-02T11:51:33.416 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T11:51:33.421 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:33.435 INFO:teuthology.orchestra.run.smithi099.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 556 kB/s | 42 kB 00:00 2024-09-02T11:51:33.448 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:33.448 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:33.448 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T11:51:33.448 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:33.448 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:33.448 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:33.486 INFO:teuthology.orchestra.run.smithi099.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 321 kB/s | 51 kB 00:00 2024-09-02T11:51:33.493 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T11:51:33.511 INFO:teuthology.orchestra.run.smithi099.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-02T11:51:33.537 INFO:teuthology.orchestra.run.smithi099.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 866 kB/s | 43 kB 00:00 2024-09-02T11:51:33.539 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T11:51:33.554 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:51:33.555 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:51:33.561 INFO:teuthology.orchestra.run.smithi099.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.5 MB/s | 172 kB 00:00 2024-09-02T11:51:33.573 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T11:51:33.587 INFO:teuthology.orchestra.run.smithi099.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 434 kB/s | 11 kB 00:00 2024-09-02T11:51:33.603 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T11:51:33.612 INFO:teuthology.orchestra.run.smithi099.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-02T11:51:33.637 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T11:51:33.637 INFO:teuthology.orchestra.run.smithi099.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-09-02T11:51:33.663 INFO:teuthology.orchestra.run.smithi099.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-02T11:51:33.679 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T11:51:33.688 INFO:teuthology.orchestra.run.smithi099.stdout:(26/37): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-09-02T11:51:33.713 INFO:teuthology.orchestra.run.smithi099.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-09-02T11:51:33.720 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T11:51:33.747 INFO:teuthology.orchestra.run.smithi099.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-09-02T11:51:33.763 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T11:51:33.772 INFO:teuthology.orchestra.run.smithi099.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-02T11:51:33.797 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T11:51:33.797 INFO:teuthology.orchestra.run.smithi099.stdout:(30/37): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-09-02T11:51:33.822 INFO:teuthology.orchestra.run.smithi099.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-02T11:51:33.838 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T11:51:33.847 INFO:teuthology.orchestra.run.smithi099.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-02T11:51:33.873 INFO:teuthology.orchestra.run.smithi099.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-02T11:51:33.878 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T11:51:33.896 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:33.898 INFO:teuthology.orchestra.run.smithi099.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-02T11:51:33.920 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:33.921 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:33.921 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T11:51:33.921 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:33.921 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:33.921 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:33.923 INFO:teuthology.orchestra.run.smithi099.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-02T11:51:33.949 INFO:teuthology.orchestra.run.smithi099.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-02T11:51:33.955 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T11:51:33.974 INFO:teuthology.orchestra.run.smithi099.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-02T11:51:33.979 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:33.979 INFO:teuthology.orchestra.run.smithi099.stdout:Total 8.8 MB/s | 31 MB 00:03 2024-09-02T11:51:34.102 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:51:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T11:51:34.119 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:34.131 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:51:34.146 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:34.146 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:34.146 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T11:51:34.146 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:34.146 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:34.147 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:34.156 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:34.170 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:51:34.171 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:51:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T11:51:34.179 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:34.179 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:34.180 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T11:51:34.180 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:34.180 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:34.180 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:34.188 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T11:51:34.215 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T11:51:34.240 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T11:51:34.242 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:34.277 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T11:51:34.336 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T11:51:34.400 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T11:51:34.467 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:51:34.467 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:51:34.594 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:34.595 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T11:51:34.618 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:34.618 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:34.618 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T11:51:34.618 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:34.618 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:34.619 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:34.648 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T11:51:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T11:51:34.819 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T11:51:34.837 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:34.855 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T11:51:34.860 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:34.860 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:34.861 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T11:51:34.861 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:34.861 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:34.861 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:34.923 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T11:51:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 32/37 2024-09-02T11:51:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:35.208 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:35.208 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T11:51:35.208 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:35.208 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:35.848 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:51:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 1/37 2024-09-02T11:51:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2/37 2024-09-02T11:51:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 3/37 2024-09-02T11:51:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 4/37 2024-09-02T11:51:35.908 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 5/37 2024-09-02T11:51:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 6/37 2024-09-02T11:51:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T11:51:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T11:51:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T11:51:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T11:51:35.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T11:51:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T11:51:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T11:51:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T11:51:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T11:51:35.910 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T11:51:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T11:51:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T11:51:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T11:51:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T11:51:35.911 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T11:51:35.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T11:51:35.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T11:51:35.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T11:51:35.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T11:51:35.912 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T11:51:35.917 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T11:51:35.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T11:51:35.918 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T11:51:35.935 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T11:51:35.976 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T11:51:36.004 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T11:51:36.075 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T11:51:36.120 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T11:51:36.152 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T11:51:36.244 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T11:51:36.314 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T11:51:36.355 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T11:51:36.418 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T11:51:36.453 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T11:51:36.488 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T11:51:36.522 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T11:51:36.561 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T11:51:36.597 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T11:51:36.631 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 1/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 3/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 4/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 5/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 6/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T11:51:36.677 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T11:51:36.678 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T11:51:36.679 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T11:51:36.679 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T11:51:36.679 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T11:51:36.679 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T11:51:36.679 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T11:51:36.679 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T11:51:36.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T11:51:36.681 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T11:51:36.712 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T11:51:36.749 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:36.776 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:36.776 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:36.776 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T11:51:36.776 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:36.777 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:36.777 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:36.784 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:36.859 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T11:51:36.860 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T11:51:36.861 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:36.862 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:36.923 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T11:51:37.010 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T11:51:37.064 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T11:51:37.120 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T11:51:37.168 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T11:51:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:37.272 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-base 2024-09-02T11:51:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:37.273 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:37.273 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T11:51:37.273 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:37.273 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:37.290 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T11:51:37.484 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T11:51:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:37.532 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T11:51:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:37.532 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:37.533 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T11:51:37.533 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:37.533 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:37.609 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T11:51:37.621 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T11:51:37.621 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:37.621 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:51:37.621 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:37.621 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:37.621 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T11:51:37.622 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T11:51:37.623 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:37.624 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:37.698 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T11:51:37.756 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T11:51:37.836 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:37.892 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T11:51:37.928 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:37.970 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:37.971 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:51:37.971 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:38.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-09-02T11:51:38.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install cephadm 2024-09-02T11:51:38.054 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 32/37 2024-09-02T11:51:38.083 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:38.108 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:38.108 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:38.108 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T11:51:38.108 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:38.108 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:38.108 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:38.593 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:51 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:38.599 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:54 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:38.683 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:38.725 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:38.726 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:51:38.726 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:38.730 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:38.730 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 782 k 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:51:38.731 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:38.732 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 782 k 2024-09-02T11:51:38.732 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 789 k 2024-09-02T11:51:38.732 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:51:38.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-09-02T11:51:39.030 INFO:teuthology.orchestra.run.smithi094.stdout:cephadm-19.3.0-4585.gb59673c4.el9.noarch.rpm 2.6 MB/s | 782 kB 00:00 2024-09-02T11:51:39.031 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:39.032 INFO:teuthology.orchestra.run.smithi094.stdout:Total 2.5 MB/s | 782 kB 00:00 2024-09-02T11:51:39.032 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:51:39.036 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:51:39.037 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:51:39.043 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:51:39.043 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:51:39.115 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:51:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 3/4 2024-09-02T11:51:39.300 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 1/37 2024-09-02T11:51:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2/37 2024-09-02T11:51:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 3/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 4/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 5/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 6/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T11:51:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T11:51:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T11:51:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T11:51:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T11:51:39.342 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:52 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:39.474 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:39.474 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:39.474 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:51:39.474 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 782 k 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 782 k 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 789 k 2024-09-02T11:51:39.475 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:51:39.515 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:39.601 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:39.618 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:39.648 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:39.648 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:39.648 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T11:51:39.648 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:39.648 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:39.648 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:39.793 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.3.0-4585.gb59673c4.el9.noarch.rpm 2.4 MB/s | 782 kB 00:00 2024-09-02T11:51:39.793 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:39.796 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.4 MB/s | 782 kB 00:00 2024-09-02T11:51:39.796 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:51:39.799 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:51:39.799 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:51:39.805 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:51:39.805 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:51:39.886 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:51:39.955 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:40.093 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:40.118 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:40.118 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T11:51:40.118 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:40.119 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:51:40.170 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:40.171 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:40.296 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:40.382 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:40.384 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:40.384 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:40.384 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T11:51:40.384 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:40.384 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:40.384 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:40.398 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T11:51:40.422 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T11:51:40.422 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:40.422 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:51:40.422 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:40.422 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T11:51:40.423 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T11:51:40.424 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:40.425 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:40.455 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:40.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-09-02T11:51:40.758 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:40.963 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:56 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:41.006 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:41.007 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:41.007 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:51:41.007 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:41.007 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:41.007 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:41.096 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:41.097 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:41.097 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-02T11:51:41.097 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:41.097 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:51:41.097 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 145 k 2024-09-02T11:51:41.097 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 145 k 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 432 k 2024-09-02T11:51:41.098 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:51:41.226 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T11:51:41.291 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-immutable-object-cache-19.3.0-4585.gb59673 751 kB/s | 145 kB 00:00 2024-09-02T11:51:41.291 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:41.292 INFO:teuthology.orchestra.run.smithi094.stdout:Total 744 kB/s | 145 kB 00:00 2024-09-02T11:51:41.292 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:51:41.300 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:51:41.300 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:51:41.304 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:41.358 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:51:41.358 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:51:41.397 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:41.440 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:41.441 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:51:41.441 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:41.500 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:51:41.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-09-02T11:51:41.559 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:41.584 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:41.584 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:41.584 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T11:51:41.584 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:41.695 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 4/4 2024-09-02T11:51:41.695 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 1/4 2024-09-02T11:51:41.695 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2/4 2024-09-02T11:51:41.695 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T11:51:41.788 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:54 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:41.919 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:41.920 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:41.920 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-02T11:51:41.920 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:41.920 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:51:41.920 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 145 k 2024-09-02T11:51:41.920 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 145 k 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 432 k 2024-09-02T11:51:41.921 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:51:41.983 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T11:51:41.983 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:41.983 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:51:41.983 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:41.984 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:41.984 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T11:51:41.984 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T11:51:41.984 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:41.984 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:51:42.066 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:54 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:42.143 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.3.0-4585.gb59673 652 kB/s | 145 kB 00:00 2024-09-02T11:51:42.144 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:42.144 INFO:teuthology.orchestra.run.smithi087.stdout:Total 647 kB/s | 145 kB 00:00 2024-09-02T11:51:42.144 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:51:42.152 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:51:42.152 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:51:42.205 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 782 k 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:42.206 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:51:42.207 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:42.207 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:51:42.207 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:42.207 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 782 k 2024-09-02T11:51:42.207 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 789 k 2024-09-02T11:51:42.207 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:51:42.209 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:51:42.209 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:51:42.256 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:42.256 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 1/37 2024-09-02T11:51:42.256 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2/37 2024-09-02T11:51:42.256 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 3/37 2024-09-02T11:51:42.256 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 4/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 5/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 6/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T11:51:42.257 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T11:51:42.258 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T11:51:42.259 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T11:51:42.261 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T11:51:42.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-09-02T11:51:42.355 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:51:42.427 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:42.453 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:42.453 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:42.453 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T11:51:42.453 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:42.460 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-19.3.0-4585.gb59673c4.el9.noarch.rpm 3.1 MB/s | 782 kB 00:00 2024-09-02T11:51:42.460 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:42.460 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.0 MB/s | 782 kB 00:00 2024-09-02T11:51:42.461 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:51:42.464 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:51:42.464 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:51:42.470 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:51:42.470 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:51:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:42.536 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:42.537 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:51:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:42.537 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:42.537 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:42.539 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:51:42.751 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr 2024-09-02T11:51:42.861 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:39 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:51:42.950 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:43.006 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 6.6 k 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 2.4 M 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 1.5 M 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 291 k 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 4.8 M 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 17 M 2024-09-02T11:51:43.009 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-02T11:51:43.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T11:51:43.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout:Install 37 Packages 2024-09-02T11:51:43.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:43.013 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 M 2024-09-02T11:51:43.013 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 109 M 2024-09-02T11:51:43.013 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:51:43.038 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:43.169 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T11:51:43.170 INFO:teuthology.orchestra.run.smithi099.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T11:51:43.171 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T11:51:43.172 INFO:teuthology.orchestra.run.smithi099.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T11:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T11:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T11:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T11:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T11:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:43.173 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:43.314 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:59 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:43.382 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:43.383 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:51:43.383 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:43.383 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:43.408 INFO:teuthology.orchestra.run.smithi094.stdout:Package ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:43.408 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:43.450 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:51:43.451 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:43.523 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-dashboard 2024-09-02T11:51:43.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2024-09-02T11:51:43.609 INFO:teuthology.orchestra.run.smithi064.stdout:(1/37): ceph-19.3.0-4585.gb59673c4.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-02T11:51:43.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-09-02T11:51:43.634 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:43.634 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:43.634 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:51:43.634 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:43.634 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:43.634 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:43.793 INFO:teuthology.orchestra.run.smithi064.stdout:(2/37): ceph-mgr-19.3.0-4585.gb59673c4.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-09-02T11:51:43.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T11:51:43.835 INFO:teuthology.orchestra.run.smithi064.stdout:(3/37): ceph-mds-19.3.0-4585.gb59673c4.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-02T11:51:43.902 INFO:teuthology.orchestra.run.smithi064.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4585.gb596 4.3 MB/s | 291 kB 00:00 2024-09-02T11:51:44.068 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:00 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:44.069 INFO:teuthology.orchestra.run.smithi064.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-02T11:51:44.114 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:44.185 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:57 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:44.186 INFO:teuthology.orchestra.run.smithi064.stdout:(6/37): ceph-mon-19.3.0-4585.gb59673c4.el9.x86_ 8.3 MB/s | 4.8 MB 00:00 2024-09-02T11:51:44.198 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:44.199 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:44.199 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:51:44.199 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:44.199 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 14 M 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 34 k 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 16 k 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-02T11:51:44.200 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-09-02T11:51:44.201 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:44.202 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 19 M 2024-09-02T11:51:44.202 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 158 M 2024-09-02T11:51:44.202 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:51:44.204 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:44.228 INFO:teuthology.orchestra.run.smithi064.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 7.7 MB/s | 1.2 MB 00:00 2024-09-02T11:51:44.246 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:44.247 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:51:44.247 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:44.278 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:44.320 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:44.321 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:51:44.321 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:44.322 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2024-09-02T11:51:44.383 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:56 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:44.390 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-09-02T11:51:44.437 INFO:teuthology.orchestra.run.smithi064.stdout:(8/37): ceph-osd-19.3.0-4585.gb59673c4.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-02T11:51:44.462 INFO:teuthology.orchestra.run.smithi064.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 579 kB/s | 135 kB 00:00 2024-09-02T11:51:44.487 INFO:teuthology.orchestra.run.smithi064.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 354 kB/s | 106 kB 00:00 2024-09-02T11:51:44.512 INFO:teuthology.orchestra.run.smithi064.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-02T11:51:44.520 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 145 k 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:51:44.521 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:44.522 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:51:44.522 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:44.522 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 145 k 2024-09-02T11:51:44.522 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 432 k 2024-09-02T11:51:44.522 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:51:44.579 INFO:teuthology.orchestra.run.smithi064.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 878 kB/s | 125 kB 00:00 2024-09-02T11:51:44.613 INFO:teuthology.orchestra.run.smithi064.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-02T11:51:44.655 INFO:teuthology.orchestra.run.smithi064.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-02T11:51:44.680 INFO:teuthology.orchestra.run.smithi064.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 345 kB/s | 35 kB 00:00 2024-09-02T11:51:44.705 INFO:teuthology.orchestra.run.smithi064.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 893 kB/s | 172 kB 00:00 2024-09-02T11:51:44.716 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-19.3.0-4585.gb59673 742 kB/s | 145 kB 00:00 2024-09-02T11:51:44.717 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:44.718 INFO:teuthology.orchestra.run.smithi038.stdout:Total 736 kB/s | 145 kB 00:00 2024-09-02T11:51:44.718 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:51:44.726 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:51:44.726 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:51:44.730 INFO:teuthology.orchestra.run.smithi064.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-02T11:51:44.756 INFO:teuthology.orchestra.run.smithi064.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-02T11:51:44.787 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:51:44.788 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:51:44.798 INFO:teuthology.orchestra.run.smithi064.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-02T11:51:44.823 INFO:teuthology.orchestra.run.smithi064.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-09-02T11:51:44.840 INFO:teuthology.orchestra.run.smithi064.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 516 kB/s | 43 kB 00:00 2024-09-02T11:51:44.864 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:44.865 INFO:teuthology.orchestra.run.smithi064.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 139 kB/s | 22 kB 00:00 2024-09-02T11:51:44.882 INFO:teuthology.orchestra.run.smithi064.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-09-02T11:51:44.892 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4585.gb5 217 kB/s | 34 kB 00:00 2024-09-02T11:51:44.907 INFO:teuthology.orchestra.run.smithi064.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-02T11:51:44.909 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4585.gb59 94 kB/s | 16 kB 00:00 2024-09-02T11:51:44.924 INFO:teuthology.orchestra.run.smithi064.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.3 MB/s | 19 kB 00:00 2024-09-02T11:51:44.936 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:57 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:44.936 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:51:44.941 INFO:teuthology.orchestra.run.smithi064.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 233 kB/s | 18 kB 00:00 2024-09-02T11:51:44.958 INFO:teuthology.orchestra.run.smithi064.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 587 kB/s | 20 kB 00:00 2024-09-02T11:51:44.975 INFO:teuthology.orchestra.run.smithi064.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-02T11:51:44.994 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:44.995 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:51:44.995 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:44.995 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:51:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 782 k 2024-09-02T11:51:44.995 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 782 k 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 789 k 2024-09-02T11:51:44.996 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:51:45.000 INFO:teuthology.orchestra.run.smithi064.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-09-02T11:51:45.001 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:45.026 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:45.026 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T11:51:45.027 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:45.027 INFO:teuthology.orchestra.run.smithi064.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-02T11:51:45.035 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-02T11:51:45.043 INFO:teuthology.orchestra.run.smithi064.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-09-02T11:51:45.067 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:45.068 INFO:teuthology.orchestra.run.smithi064.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-02T11:51:45.068 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:45.068 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:51:45.068 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 14 M 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 34 k 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 16 k 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-02T11:51:45.069 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-09-02T11:51:45.070 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:45.071 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 19 M 2024-09-02T11:51:45.071 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 158 M 2024-09-02T11:51:45.071 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:51:45.093 INFO:teuthology.orchestra.run.smithi064.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-02T11:51:45.119 INFO:teuthology.orchestra.run.smithi064.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-02T11:51:45.144 INFO:teuthology.orchestra.run.smithi064.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-02T11:51:45.152 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-09-02T11:51:45.177 INFO:teuthology.orchestra.run.smithi064.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-02T11:51:45.211 INFO:teuthology.orchestra.run.smithi064.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-02T11:51:45.215 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:45.215 INFO:teuthology.orchestra.run.smithi064.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-02T11:51:45.219 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-02T11:51:45.244 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-09-02T11:51:45.369 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:51:45.384 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-19.3.0-4585.gb59673c4.el9.noarch.rpm 2.0 MB/s | 782 kB 00:00 2024-09-02T11:51:45.384 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:45.385 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.0 MB/s | 782 kB 00:00 2024-09-02T11:51:45.385 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:51:45.390 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:51:45.390 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:51:45.396 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:51:45.397 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:51:45.411 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:51:45.411 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:51:45.428 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4585.gb59673c 20 MB/s | 14 MB 00:00 2024-09-02T11:51:45.464 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:51:45.470 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-09-02T11:51:45.520 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-09-02T11:51:45.545 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 419 kB/s | 31 kB 00:00 2024-09-02T11:51:45.570 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.8 MB/s | 188 kB 00:00 2024-09-02T11:51:45.612 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.5 MB/s | 2.0 MB 00:00 2024-09-02T11:51:45.616 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:45.616 INFO:teuthology.orchestra.run.smithi094.stdout:Total 14 MB/s | 19 MB 00:01 2024-09-02T11:51:45.662 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:51:45.680 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:51:45.680 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:51:45.723 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:51:45.728 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:51:45.874 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:45.875 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:45.875 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:51:45.875 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:45.875 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:45.875 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:45.902 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:51:45.902 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:51:45.920 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:45.930 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4585.gb5 221 kB/s | 34 kB 00:00 2024-09-02T11:51:45.955 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4585.gb59 91 kB/s | 16 kB 00:00 2024-09-02T11:51:46.015 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:46.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-09-02T11:51:46.353 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:46.455 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4585.gb59673c 21 MB/s | 14 MB 00:00 2024-09-02T11:51:46.509 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:51:46.588 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-02T11:51:46.595 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:46.595 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:46.595 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:51:46.595 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:46.596 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:46.596 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:46.630 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-02T11:51:46.631 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-02T11:51:46.641 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:46.677 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-02T11:51:46.680 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-02T11:51:46.706 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-02T11:51:46.717 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-02T11:51:46.735 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:46.779 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:46.780 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:51:46.780 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:46.781 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-02T11:51:46.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T11:51:46.792 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-02T11:51:46.806 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-02T11:51:46.831 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-02T11:51:46.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-09-02T11:51:46.857 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-02T11:51:46.882 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-protobuf-3.14.0-13.el9.noarch. 290 kB/s | 269 kB 00:00 2024-09-02T11:51:46.918 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-02T11:51:47.011 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-02T11:51:47.032 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 937 kB/s | 1.0 MB 00:01 2024-09-02T11:51:47.036 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:47.036 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.8 MB/s | 19 MB 00:01 2024-09-02T11:51:47.072 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-02T11:51:47.085 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:51:47.101 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:47.103 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:51:47.104 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:51:47.133 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 10/12 2024-09-02T11:51:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:51:47.272 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T11:51:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T11:51:47.327 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:51:47.328 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:51:47.341 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T11:51:47.349 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:56 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:47.398 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:59 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:47.415 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T11:51:47.457 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T11:51:47.482 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 145 k 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:51:47.483 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:47.484 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 145 k 2024-09-02T11:51:47.484 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 432 k 2024-09-02T11:51:47.484 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:51:47.507 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T11:51:47.535 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:47.536 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 14 M 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 34 k 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 16 k 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-02T11:51:47.537 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-09-02T11:51:47.538 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:47.539 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 19 M 2024-09-02T11:51:47.539 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 158 M 2024-09-02T11:51:47.539 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:51:47.603 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T11:51:47.680 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T11:51:47.683 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-19.3.0-4585.gb59673 725 kB/s | 145 kB 00:00 2024-09-02T11:51:47.684 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:47.685 INFO:teuthology.orchestra.run.smithi099.stdout:Total 718 kB/s | 145 kB 00:00 2024-09-02T11:51:47.685 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:51:47.692 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:51:47.692 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:51:47.735 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T11:51:47.749 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:51:47.750 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:51:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T11:51:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T11:51:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T11:51:47.867 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T11:51:47.899 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:51:47.907 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T11:51:47.921 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:51:47.968 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T11:51:47.976 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:47.999 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-02T11:51:48.001 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:48.001 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:48.001 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T11:51:48.001 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T11:51:48.034 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-02T11:51:48.053 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4585.gb5 231 kB/s | 34 kB 00:00 2024-09-02T11:51:48.059 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T11:51:48.079 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4585.gb59 95 kB/s | 16 kB 00:00 2024-09-02T11:51:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-02T11:51:48.099 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T11:51:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-02T11:51:48.176 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T11:51:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-02T11:51:48.221 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-09-02T11:51:48.272 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-02T11:51:48.322 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-02T11:51:48.346 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-02T11:51:48.347 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-09-02T11:51:48.439 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-02T11:51:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T11:51:48.481 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-09-02T11:51:48.491 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-02T11:51:48.506 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-02T11:51:48.520 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:48.531 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-02T11:51:48.534 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T11:51:48.552 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 10/12 2024-09-02T11:51:48.557 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-02T11:51:48.586 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T11:51:48.623 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T11:51:48.682 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T11:51:48.707 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4585.gb59673 18 MB/s | 14 MB 00:00 2024-09-02T11:51:48.732 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.7 MB/s | 863 kB 00:00 2024-09-02T11:51:48.735 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:48.735 INFO:teuthology.orchestra.run.smithi038.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-02T11:51:48.746 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T11:51:48.784 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:51:48.804 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:51:48.804 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:51:48.939 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T11:51:48.960 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:48.960 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:48.960 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:51:48.960 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:48.960 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:48.961 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:49.028 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T11:51:49.035 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:51:49.036 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:51:49.108 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T11:51:49.194 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2024-09-02T11:51:49.322 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T11:51:49.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T11:51:49.496 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T11:51:49.637 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:51:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 32/37 2024-09-02T11:51:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-02T11:51:49.720 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 33/37 2024-09-02T11:51:49.720 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:49.720 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T11:51:49.720 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:49.721 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T11:51:49.721 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-02T11:51:49.766 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-02T11:51:49.822 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-02T11:51:49.856 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:49.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-02T11:51:49.898 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:49.899 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:51:49.899 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:49.973 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-dashboard 2024-09-02T11:51:50.029 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-02T11:51:50.130 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-02T11:51:50.178 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-02T11:51:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:50.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 10/12 2024-09-02T11:51:50.331 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 11/12 2024-09-02T11:51:50.388 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:50.524 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:59 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:51:50.656 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:51:50.657 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 14 M 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 34 k 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 16 k 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-02T11:51:50.658 INFO:teuthology.orchestra.run.smithi099.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout:Install 12 Packages 2024-09-02T11:51:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:50.660 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 19 M 2024-09-02T11:51:50.660 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 158 M 2024-09-02T11:51:50.660 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:51:51.262 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 34/37 2024-09-02T11:51:51.289 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:51.290 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T11:51:51.290 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:51.290 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T11:51:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:51.478 INFO:teuthology.orchestra.run.smithi099.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4585.gb5 233 kB/s | 34 kB 00:00 2024-09-02T11:51:51.520 INFO:teuthology.orchestra.run.smithi099.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4585.gb59 87 kB/s | 16 kB 00:00 2024-09-02T11:51:51.712 INFO:teuthology.orchestra.run.smithi099.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-02T11:51:51.748 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 11/12 2024-09-02T11:51:51.754 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:51.778 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 35/37 2024-09-02T11:51:51.778 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:51.778 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T11:51:51.778 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:51.778 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T11:51:51.778 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:51.807 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:52.004 INFO:teuthology.orchestra.run.smithi099.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4585.gb59673c 21 MB/s | 14 MB 00:00 2024-09-02T11:51:52.013 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:52.037 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 36/37 2024-09-02T11:51:52.037 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:52.037 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T11:51:52.037 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:52.037 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T11:51:52.037 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:52.038 INFO:teuthology.orchestra.run.smithi099.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-02T11:51:52.079 INFO:teuthology.orchestra.run.smithi099.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-09-02T11:51:52.103 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:52.112 INFO:teuthology.orchestra.run.smithi099.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-09-02T11:51:52.154 INFO:teuthology.orchestra.run.smithi099.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.6 MB/s | 548 kB 00:00 2024-09-02T11:51:52.179 INFO:teuthology.orchestra.run.smithi099.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-02T11:51:52.205 INFO:teuthology.orchestra.run.smithi099.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.5 MB/s | 144 kB 00:00 2024-09-02T11:51:52.230 INFO:teuthology.orchestra.run.smithi099.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-02T11:51:52.280 INFO:teuthology.orchestra.run.smithi099.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 10 MB/s | 2.0 MB 00:00 2024-09-02T11:51:52.284 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:52.285 INFO:teuthology.orchestra.run.smithi099.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-02T11:51:52.336 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:51:52.355 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:51:52.355 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:51:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 1/12 2024-09-02T11:51:52.453 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 2/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 3/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-02T11:51:52.454 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-02T11:51:52.580 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:51:52.581 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:51:52.759 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-02T11:51:52.759 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-02T11:51:52.760 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-02T11:51:52.761 INFO:teuthology.orchestra.run.smithi094.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-02T11:51:52.761 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-02T11:51:52.761 INFO:teuthology.orchestra.run.smithi094.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-02T11:51:52.761 INFO:teuthology.orchestra.run.smithi094.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-02T11:51:52.762 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-02T11:51:52.762 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-02T11:51:52.762 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:52.762 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:51:53.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-02T11:51:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:51:53.255 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-02T11:51:53.291 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-02T11:51:53.337 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-02T11:51:53.376 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-02T11:51:53.453 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-02T11:51:53.554 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:09 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:51:53.575 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-02T11:51:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 11/12 2024-09-02T11:51:53.664 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-02T11:51:53.667 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:53.686 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:51:53.687 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-09-02T11:51:53.687 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 7.4 M 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-02T11:51:53.688 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout:======================================================================================= 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout:Install 12 Packages 2024-09-02T11:51:53.689 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:51:53.690 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 38 M 2024-09-02T11:51:53.690 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 212 M 2024-09-02T11:51:53.690 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 1/12 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 2/12 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 3/12 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-02T11:51:53.701 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-02T11:51:53.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-02T11:51:53.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-02T11:51:53.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:53.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-02T11:51:53.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-02T11:51:53.706 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-02T11:51:53.734 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:53.766 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 10/12 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 37/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 1/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 3/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 4/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 5/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9. 6/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T11:51:53.818 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T11:51:53.819 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T11:51:53.820 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T11:51:53.820 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T11:51:53.820 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T11:51:53.820 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T11:51:53.820 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T11:51:53.820 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T11:51:53.821 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T11:51:53.821 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T11:51:53.821 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T11:51:53.821 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T11:51:53.821 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T11:51:53.822 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T11:51:53.822 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T11:51:53.822 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T11:51:53.822 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-02T11:51:54.005 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-02T11:51:54.006 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-02T11:51:54.006 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-02T11:51:54.006 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-02T11:51:54.006 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:54.006 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:51:54.221 INFO:teuthology.orchestra.run.smithi094.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-02T11:51:54.258 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-02T11:51:54.262 INFO:teuthology.orchestra.run.smithi094.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-02T11:51:54.563 INFO:teuthology.orchestra.run.smithi094.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-45 14 MB/s | 7.4 MB 00:00 2024-09-02T11:51:54.805 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:07 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:51:54.822 INFO:teuthology.orchestra.run.smithi094.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-02T11:51:54.927 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T11:51:54.927 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:54.927 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:51:54.927 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:54.927 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T11:51:54.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T11:51:54.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:54.930 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:51:54.935 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:51:54.936 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 7.4 M 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-02T11:51:54.937 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-09-02T11:51:54.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:51:54.939 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 38 M 2024-09-02T11:51:54.939 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 212 M 2024-09-02T11:51:54.939 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:51:55.047 INFO:teuthology.orchestra.run.smithi094.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-02T11:51:55.213 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-02T11:51:55.255 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-02T11:51:55.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-09-02T11:51:55.355 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-09-02T11:51:55.397 INFO:teuthology.orchestra.run.smithi094.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-02T11:51:55.406 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-02T11:51:55.422 INFO:teuthology.orchestra.run.smithi094.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 64 kB/s | 38 kB 00:00 2024-09-02T11:51:55.464 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 661 kB/s | 38 kB 00:00 2024-09-02T11:51:55.525 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:55.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 1/12 2024-09-02T11:51:55.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 2/12 2024-09-02T11:51:55.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 3/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-02T11:51:55.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-02T11:51:55.540 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-02T11:51:55.590 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-02T11:51:55.682 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): ceph-mgr-diskprediction-local-19.3.0-45 12 MB/s | 7.4 MB 00:00 2024-09-02T11:51:55.774 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-09-02T11:51:55.836 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-02T11:51:55.837 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-02T11:51:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-02T11:51:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-02T11:51:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-02T11:51:55.838 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:55.838 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:51:55.840 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 13 MB/s | 4.9 MB 00:00 2024-09-02T11:51:55.893 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:52 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:51:55.941 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 969 kB/s | 250 kB 00:00 2024-09-02T11:51:55.989 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:51:56.032 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:51:56.033 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:51:56.033 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:51:56.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-02T11:51:56.108 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-02T11:51:56.109 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:56.109 INFO:teuthology.orchestra.run.smithi087.stdout:Total 33 MB/s | 38 MB 00:01 2024-09-02T11:51:56.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-09-02T11:51:56.148 INFO:teuthology.orchestra.run.smithi094.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 327 kB/s | 245 kB 00:00 2024-09-02T11:51:56.223 INFO:teuthology.orchestra.run.smithi094.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.2 MB/s | 4.9 MB 00:01 2024-09-02T11:51:56.464 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:51:56.489 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:51:56.489 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:51:56.607 INFO:teuthology.orchestra.run.smithi094.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-09-02T11:51:56.629 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:51:56.671 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:51:56.766 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:51:56.767 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-09-02T11:51:56.767 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:51:56.767 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 7.4 M 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-02T11:51:56.768 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-09-02T11:51:56.769 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-09-02T11:51:56.770 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:51:56.770 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 38 M 2024-09-02T11:51:56.770 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 212 M 2024-09-02T11:51:56.770 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:51:56.809 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:51:56.809 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:51:56.809 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 782 k 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 782 k 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 789 k 2024-09-02T11:51:56.810 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:51:56.899 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:51:56.899 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:51:56.915 INFO:teuthology.orchestra.run.smithi094.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 327 kB/s | 250 kB 00:00 2024-09-02T11:51:56.947 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 11/12 2024-09-02T11:51:57.004 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:57.273 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.3.0-4585.gb59673c4.el9.noarch.rpm 2.7 MB/s | 782 kB 00:00 2024-09-02T11:51:57.273 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:57.273 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.6 MB/s | 782 kB 00:00 2024-09-02T11:51:57.273 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:51:57.273 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:51:57.274 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:51:57.274 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:51:57.274 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:51:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:51:57.449 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-02T11:51:57.490 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 789 kB/s | 33 kB 00:00 2024-09-02T11:51:57.524 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-02T11:51:57.566 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-02T11:51:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:57.608 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 929 kB/s | 38 kB 00:00 2024-09-02T11:51:57.657 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:51:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:57.682 INFO:teuthology.orchestra.run.smithi094.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-02T11:51:57.683 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:57.684 INFO:teuthology.orchestra.run.smithi094.stdout:Total 9.6 MB/s | 38 MB 00:03 2024-09-02T11:51:57.720 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-02T11:51:57.764 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:51:57.850 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-45 14 MB/s | 7.4 MB 00:00 2024-09-02T11:51:57.861 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:51:57.915 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-02T11:51:57.967 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-02T11:51:58.003 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:58.042 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:51:58.067 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:51:58.068 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:51:58.099 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-02T11:51:58.131 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-02T11:51:58.184 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2024-09-02T11:51:58.207 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-02T11:51:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:51:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:58.213 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:51:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:58.213 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:51:58.276 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-02T11:51:58.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T11:51:58.478 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:51:58.478 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 1/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 2/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 3/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-02T11:51:58.845 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-02T11:51:58.847 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-02T11:51:58.968 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-02T11:51:58.980 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:55 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:51:58.992 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.6 MB/s | 4.9 MB 00:01 2024-09-02T11:51:59.115 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:51:59.116 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:59.116 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 145 k 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 432 k 2024-09-02T11:51:59.117 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:51:59.168 INFO:teuthology.orchestra.run.smithi099.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:51:59.169 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:51:59.201 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-09-02T11:51:59.239 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:51:59.311 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-02T11:51:59.331 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.3.0-4585.gb59673 678 kB/s | 145 kB 00:00 2024-09-02T11:51:59.331 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:51:59.332 INFO:teuthology.orchestra.run.smithi064.stdout:Total 673 kB/s | 145 kB 00:00 2024-09-02T11:51:59.332 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:51:59.341 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:51:59.341 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:51:59.355 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:51:59.401 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:51:59.402 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:51:59.432 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-02T11:51:59.456 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:51:59.506 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-02T11:51:59.544 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:51:59.624 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:51:59.649 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:51:59.649 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T11:51:59.649 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:51:59.690 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-02T11:51:59.722 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-02T11:51:59.798 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-02T11:51:59.986 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:00.119 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:00.120 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================= 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================= 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 7.4 M 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-02T11:52:00.121 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================= 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout:Install 12 Packages 2024-09-02T11:52:00.122 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:00.123 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 38 M 2024-09-02T11:52:00.123 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 212 M 2024-09-02T11:52:00.123 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:00.223 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:00.313 INFO:teuthology.orchestra.run.smithi099.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-02T11:52:00.354 INFO:teuthology.orchestra.run.smithi099.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 786 kB/s | 33 kB 00:00 2024-09-02T11:52:00.380 INFO:teuthology.orchestra.run.smithi099.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-02T11:52:00.422 INFO:teuthology.orchestra.run.smithi099.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-02T11:52:00.438 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-02T11:52:00.464 INFO:teuthology.orchestra.run.smithi099.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 927 kB/s | 38 kB 00:00 2024-09-02T11:52:00.541 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4. 1/1 2024-09-02T11:52:00.541 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:00.541 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:00.541 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:00.541 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:00.541 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:00.568 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-02T11:52:00.664 INFO:teuthology.orchestra.run.smithi099.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-45 16 MB/s | 7.4 MB 00:00 2024-09-02T11:52:00.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-09-02T11:52:00.814 INFO:teuthology.orchestra.run.smithi099.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-02T11:52:01.090 INFO:teuthology.orchestra.run.smithi099.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-09-02T11:52:01.181 INFO:teuthology.orchestra.run.smithi099.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-02T11:52:01.249 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-02T11:52:01.280 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:01.322 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:01.415 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:01.459 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:01.460 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:01.460 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:01.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-09-02T11:52:01.784 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:01.815 INFO:teuthology.orchestra.run.smithi099.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.7 MB/s | 4.9 MB 00:01 2024-09-02T11:52:02.110 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:59 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:02.173 INFO:teuthology.orchestra.run.smithi099.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-02T11:52:02.246 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 14 M 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 34 k 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 16 k 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-02T11:52:02.248 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-02T11:52:02.249 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:02.250 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 19 M 2024-09-02T11:52:02.250 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 158 M 2024-09-02T11:52:02.250 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:02.609 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.8 MB/s | 16 MB 00:04 2024-09-02T11:52:02.610 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:02.610 INFO:teuthology.orchestra.run.smithi038.stdout:Total 6.5 MB/s | 38 MB 00:05 2024-09-02T11:52:02.829 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-02T11:52:02.856 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:02.953 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4585.gb5 247 kB/s | 34 kB 00:00 2024-09-02T11:52:02.978 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:02.979 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4585.gb59 101 kB/s | 16 kB 00:00 2024-09-02T11:52:03.005 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:03.005 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:03.203 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-02T11:52:03.295 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-09-02T11:52:03.440 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:03.440 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:03.454 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4585.gb59673c 22 MB/s | 14 MB 00:00 2024-09-02T11:52:03.488 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-09-02T11:52:03.538 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 239 kB/s | 19 kB 00:00 2024-09-02T11:52:03.571 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-09-02T11:52:03.596 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.5 MB/s | 144 kB 00:00 2024-09-02T11:52:03.623 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 613 kB/s | 31 kB 00:00 2024-09-02T11:52:03.655 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-02T11:52:03.723 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.0 MB/s | 2.0 MB 00:00 2024-09-02T11:52:03.726 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:03.727 INFO:teuthology.orchestra.run.smithi064.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 1/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-02T11:52:03.777 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-02T11:52:03.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-02T11:52:03.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:03.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-02T11:52:03.778 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:03.795 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:03.795 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:04.027 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:04.027 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:04.179 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-02T11:52:04.179 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-02T11:52:04.180 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:04.181 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:04.181 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-02T11:52:04.181 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:04.181 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:04.306 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-02T11:52:04.358 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:04.470 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:04.509 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-02T11:52:04.563 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-09-02T11:52:04.701 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-02T11:52:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-02T11:52:04.750 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:04.820 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-02T11:52:04.832 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-02T11:52:04.875 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-02T11:52:04.923 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-02T11:52:04.952 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-02T11:52:05.032 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-02T11:52:05.123 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:18 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-02T11:52:05.256 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:05.257 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:05.257 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 49 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-02T11:52:05.258 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-09-02T11:52:05.259 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-02T11:52:05.260 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.2 M 2024-09-02T11:52:05.260 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-09-02T11:52:05.260 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-02T11:52:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:52:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 10/12 2024-09-02T11:52:05.473 INFO:teuthology.orchestra.run.smithi099.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.8 MB/s | 16 MB 00:04 2024-09-02T11:52:05.474 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:05.474 INFO:teuthology.orchestra.run.smithi099.stdout:Total 7.1 MB/s | 38 MB 00:05 2024-09-02T11:52:05.496 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 1/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-02T11:52:05.546 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:05.547 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-02T11:52:05.561 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 220 kB/s | 26 kB 00:00 2024-09-02T11:52:05.578 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 144 kB/s | 19 kB 00:00 2024-09-02T11:52:05.595 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): ceph-mgr-rook-19.3.0-4585.gb59673c4.el9 328 kB/s | 49 kB 00:00 2024-09-02T11:52:05.618 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-02T11:52:05.704 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-02T11:52:05.729 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-02T11:52:05.754 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-02T11:52:05.779 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 640 kB/s | 32 kB 00:00 2024-09-02T11:52:05.805 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 291 kB/s | 14 kB 00:00 2024-09-02T11:52:05.830 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-02T11:52:05.837 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:05.855 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-02T11:52:05.863 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:05.864 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:05.889 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.4 MB/s | 90 kB 00:00 2024-09-02T11:52:05.922 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-02T11:52:05.926 INFO:teuthology.orchestra.run.smithi094.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-02T11:52:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-02T11:52:05.927 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:05.927 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:05.948 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-09-02T11:52:05.950 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:05.950 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-02T11:52:06.012 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:06.028 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:06.028 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:06.115 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:06.115 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:06.280 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:06.280 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:06.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-rook 2024-09-02T11:52:06.648 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:06.820 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-02T11:52:06.873 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:22 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-02T11:52:06.910 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-02T11:52:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-02T11:52:07.007 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 49 k 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-02T11:52:07.008 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:07.009 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:07.010 INFO:teuthology.orchestra.run.smithi094.stdout:Install 13 Packages 2024-09-02T11:52:07.010 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:07.010 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 2.2 M 2024-09-02T11:52:07.010 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 26 M 2024-09-02T11:52:07.010 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:07.025 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-02T11:52:07.070 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:07.133 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-02T11:52:07.136 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-02T11:52:07.171 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:07.236 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-02T11:52:07.266 INFO:teuthology.orchestra.run.smithi094.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 262 kB/s | 26 kB 00:00 2024-09-02T11:52:07.277 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:07.291 INFO:teuthology.orchestra.run.smithi094.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 156 kB/s | 19 kB 00:00 2024-09-02T11:52:07.300 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-02T11:52:07.322 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-02T11:52:07.349 INFO:teuthology.orchestra.run.smithi094.stdout:(3/13): ceph-mgr-rook-19.3.0-4585.gb59673c4.el9 271 kB/s | 49 kB 00:00 2024-09-02T11:52:07.400 INFO:teuthology.orchestra.run.smithi094.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-02T11:52:07.425 INFO:teuthology.orchestra.run.smithi094.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-02T11:52:07.450 INFO:teuthology.orchestra.run.smithi094.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-02T11:52:07.492 INFO:teuthology.orchestra.run.smithi094.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-02T11:52:07.505 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-02T11:52:07.520 INFO:teuthology.orchestra.run.smithi094.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 234 kB/s | 14 kB 00:00 2024-09-02T11:52:07.529 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-02T11:52:07.542 INFO:teuthology.orchestra.run.smithi094.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-02T11:52:07.568 INFO:teuthology.orchestra.run.smithi094.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-02T11:52:07.593 INFO:teuthology.orchestra.run.smithi094.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-02T11:52:07.613 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-02T11:52:07.644 INFO:teuthology.orchestra.run.smithi094.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-02T11:52:07.669 INFO:teuthology.orchestra.run.smithi094.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 893 kB/s | 90 kB 00:00 2024-09-02T11:52:07.671 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:07.671 INFO:teuthology.orchestra.run.smithi094.stdout:Total 3.3 MB/s | 2.2 MB 00:00 2024-09-02T11:52:07.735 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:07.750 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:07.750 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:07.837 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:07.839 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:07.994 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-02T11:52:08.017 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-02T11:52:08.023 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-02T11:52:08.107 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-02T11:52:08.155 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-02T11:52:08.177 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:08.251 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-02T11:52:08.342 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:08.383 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-02T11:52:08.508 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-02T11:52:08.570 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-02T11:52:08.634 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-02T11:52:08.671 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-02T11:52:08.713 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-02T11:52:08.751 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 11/12 2024-09-02T11:52:08.805 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:52:08.815 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-02T11:52:08.916 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-02T11:52:08.947 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:08.948 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 1/13 2024-09-02T11:52:08.948 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-02T11:52:08.948 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-02T11:52:08.948 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-02T11:52:08.948 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-02T11:52:08.949 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-02T11:52:08.949 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-02T11:52:08.949 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-02T11:52:08.949 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-02T11:52:08.949 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-02T11:52:08.950 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-02T11:52:08.950 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-02T11:52:08.974 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-02T11:52:09.268 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-02T11:52:09.268 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:09.268 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:09.268 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:09.268 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-02T11:52:09.269 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-02T11:52:09.270 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-02T11:52:09.270 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:09.270 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:09.539 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-09-02T11:52:09.605 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:09.685 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-02T11:52:09.737 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-02T11:52:09.771 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-02T11:52:09.819 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-02T11:52:09.849 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:10.094 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:23 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:10.230 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:10.230 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 154 k 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:10.231 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-09-02T11:52:10.232 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 7.0 M 2024-09-02T11:52:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-09-02T11:52:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:10.525 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:10.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 1/12 2024-09-02T11:52:10.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:10.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:10.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-02T11:52:10.525 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-02T11:52:10.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-02T11:52:10.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-02T11:52:10.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-02T11:52:10.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-02T11:52:10.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:10.526 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-02T11:52:10.558 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-09-02T11:52:10.608 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-02T11:52:10.627 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-02T11:52:10.632 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:10.632 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 1/13 2024-09-02T11:52:10.632 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-02T11:52:10.632 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-02T11:52:10.633 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-02T11:52:10.641 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4585.gb59673c4.e 639 kB/s | 154 kB 00:00 2024-09-02T11:52:10.653 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:10.667 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 985 kB/s | 58 kB 00:00 2024-09-02T11:52:10.709 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-02T11:52:10.753 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 12/12 2024-09-02T11:52:10.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el 1/12 2024-09-02T11:52:10.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noa 2/12 2024-09-02T11:52:10.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9 3/12 2024-09-02T11:52:10.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-02T11:52:10.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-02T11:52:10.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-02T11:52:10.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-02T11:52:10.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-02T11:52:10.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-02T11:52:10.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-02T11:52:10.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-02T11:52:10.784 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-02T11:52:10.786 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:10.787 INFO:teuthology.orchestra.run.smithi087.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-02T11:52:10.859 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:10.870 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:10.870 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:10.926 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:10.927 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:10.928 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-02T11:52:10.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:10.928 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:10.972 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:10.972 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-02T11:52:10.978 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:10.979 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-02T11:52:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:11.063 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:11.205 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-mgr-cephadm 2024-09-02T11:52:11.294 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:11.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-09-02T11:52:11.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-02T11:52:11.466 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-02T11:52:11.536 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-02T11:52:11.775 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:27 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:11.867 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:11.875 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:23 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:11.909 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:11.909 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 154 k 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-02T11:52:11.910 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout:Install 6 Packages 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 7.0 M 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 33 M 2024-09-02T11:52:11.911 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:12.004 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:12.005 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-02T11:52:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:52:12.005 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-02T11:52:12.005 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 7.4 M 2024-09-02T11:52:12.005 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-02T11:52:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-02T11:52:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:12.008 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-09-02T11:52:12.008 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 212 M 2024-09-02T11:52:12.008 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:12.013 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 49 k 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-02T11:52:12.015 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:12.016 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-09-02T11:52:12.017 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:12.017 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.2 M 2024-09-02T11:52:12.017 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-09-02T11:52:12.017 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:12.053 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-02T11:52:12.138 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-02T11:52:12.156 INFO:teuthology.orchestra.run.smithi094.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-02T11:52:12.206 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-02T11:52:12.216 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 156 kB/s | 19 kB 00:00 2024-09-02T11:52:12.231 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:12.241 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 26 kB 00:00 2024-09-02T11:52:12.266 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): ceph-mgr-rook-19.3.0-4585.gb59673c4.el9 284 kB/s | 49 kB 00:00 2024-09-02T11:52:12.281 INFO:teuthology.orchestra.run.smithi094.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-09-02T11:52:12.306 INFO:teuthology.orchestra.run.smithi094.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4585.gb59673c4.e 533 kB/s | 154 kB 00:00 2024-09-02T11:52:12.348 INFO:teuthology.orchestra.run.smithi094.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2024-09-02T11:52:12.376 INFO:teuthology.orchestra.run.smithi094.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 627 kB/s | 58 kB 00:00 2024-09-02T11:52:12.392 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-02T11:52:12.399 INFO:teuthology.orchestra.run.smithi094.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 873 kB/s | 74 kB 00:00 2024-09-02T11:52:12.403 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:12.403 INFO:teuthology.orchestra.run.smithi094.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-02T11:52:12.417 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 222 kB 00:00 2024-09-02T11:52:12.451 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 912 kB/s | 54 kB 00:00 2024-09-02T11:52:12.473 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:12.484 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-02T11:52:12.485 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:12.485 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:12.509 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-02T11:52:12.534 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-02T11:52:12.585 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:12.585 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:12.585 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.3 MB/s | 59 kB 00:00 2024-09-02T11:52:12.598 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-45 15 MB/s | 7.4 MB 00:00 2024-09-02T11:52:12.610 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.7 MB/s | 220 kB 00:00 2024-09-02T11:52:12.644 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.6 MB/s | 90 kB 00:00 2024-09-02T11:52:12.702 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-09-02T11:52:12.704 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 311 kB/s | 33 kB 00:00 2024-09-02T11:52:12.705 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:12.705 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-02T11:52:12.770 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:12.788 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:12.788 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:12.875 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:12.876 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:12.914 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:12.963 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 221 kB/s | 191 kB 00:00 2024-09-02T11:52:13.030 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 860 kB/s | 801 kB 00:00 2024-09-02T11:52:13.033 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:13.033 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/6 2024-09-02T11:52:13.034 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-02T11:52:13.034 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-02T11:52:13.034 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-02T11:52:13.034 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-02T11:52:13.055 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 199 kB/s | 18 kB 00:00 2024-09-02T11:52:13.089 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-02T11:52:13.139 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 355 kB/s | 38 kB 00:00 2024-09-02T11:52:13.142 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-02T11:52:13.281 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-02T11:52:13.292 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-02T11:52:13.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:13.293 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:13.306 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:13.306 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 1/12 2024-09-02T11:52:13.306 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:13.306 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:13.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-02T11:52:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:13.556 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-09-02T11:52:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-02T11:52:13.606 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.3 MB/s | 3.0 MB 00:00 2024-09-02T11:52:13.614 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-02T11:52:13.669 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-02T11:52:13.676 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-02T11:52:13.689 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.0 MB/s | 250 kB 00:00 2024-09-02T11:52:13.696 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:13.697 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:13.698 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-02T11:52:13.698 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:13.698 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:13.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-02T11:52:13.747 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-02T11:52:13.770 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-02T11:52:13.838 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-02T11:52:13.853 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-02T11:52:13.863 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-02T11:52:14.023 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-02T11:52:14.104 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:27 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:14.115 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-rook 2024-09-02T11:52:14.236 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 900 k 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-02T11:52:14.237 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 979 k 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.9 M 2024-09-02T11:52:14.238 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:14.373 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.7 MB/s | 4.9 MB 00:01 2024-09-02T11:52:14.486 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 521 kB/s | 80 kB 00:00 2024-09-02T11:52:14.598 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.9 MB/s | 5.1 MB 00:01 2024-09-02T11:52:14.611 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.3.0-4585.gb59673c4.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-02T11:52:14.611 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:14.611 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.6 MB/s | 979 kB 00:00 2024-09-02T11:52:14.623 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:14.645 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:14.645 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/6 2024-09-02T11:52:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-02T11:52:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-02T11:52:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-02T11:52:14.651 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-02T11:52:14.684 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:23 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:14.724 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:14.724 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:14.784 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-02T11:52:14.815 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:14.816 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:14.816 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:52:14.816 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:14.816 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:14.816 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 49 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-02T11:52:14.817 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:Install 13 Packages 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 2.2 M 2024-09-02T11:52:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 26 M 2024-09-02T11:52:14.819 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:14.837 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-02T11:52:14.871 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-02T11:52:14.909 INFO:teuthology.orchestra.run.smithi094.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-02T11:52:14.910 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:14.910 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:14.918 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:14.919 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-02T11:52:14.948 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:15.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T11:52:15.069 INFO:teuthology.orchestra.run.smithi099.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 181 kB/s | 19 kB 00:00 2024-09-02T11:52:15.095 INFO:teuthology.orchestra.run.smithi099.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 26 kB 00:00 2024-09-02T11:52:15.121 INFO:teuthology.orchestra.run.smithi099.stdout:(3/13): ceph-mgr-rook-19.3.0-4585.gb59673c4.el9 315 kB/s | 49 kB 00:00 2024-09-02T11:52:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:15.141 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-fuse 2024-09-02T11:52:15.179 INFO:teuthology.orchestra.run.smithi099.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-02T11:52:15.204 INFO:teuthology.orchestra.run.smithi099.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-02T11:52:15.229 INFO:teuthology.orchestra.run.smithi099.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-02T11:52:15.271 INFO:teuthology.orchestra.run.smithi099.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-09-02T11:52:15.305 INFO:teuthology.orchestra.run.smithi099.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-02T11:52:15.338 INFO:teuthology.orchestra.run.smithi099.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 130 kB/s | 14 kB 00:00 2024-09-02T11:52:15.372 INFO:teuthology.orchestra.run.smithi099.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-02T11:52:15.397 INFO:teuthology.orchestra.run.smithi099.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-02T11:52:15.423 INFO:teuthology.orchestra.run.smithi099.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-02T11:52:15.473 INFO:teuthology.orchestra.run.smithi099.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.3 MB/s | 1.0 MB 00:00 2024-09-02T11:52:15.475 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:15.476 INFO:teuthology.orchestra.run.smithi099.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-02T11:52:15.540 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:15.555 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:15.555 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:15.642 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:15.643 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 1/13 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-02T11:52:15.678 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-02T11:52:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-02T11:52:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-02T11:52:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-02T11:52:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-02T11:52:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-02T11:52:15.679 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-02T11:52:15.699 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:31 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:15.833 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 900 k 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout:Install 2 Packages 2024-09-02T11:52:15.834 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:15.835 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 979 k 2024-09-02T11:52:15.835 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 2.9 M 2024-09-02T11:52:15.835 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:15.840 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:15.840 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/2 2024-09-02T11:52:16.094 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T11:52:16.094 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:16.094 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:16.095 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T11:52:16.095 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:16.095 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:16.099 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-02T11:52:16.099 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:16.099 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:16.099 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-02T11:52:16.100 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-02T11:52:16.101 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:16.101 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:16.144 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:16.193 INFO:teuthology.orchestra.run.smithi094.stdout:(1/2): ceph-fuse-19.3.0-4585.gb59673c4.el9.x86_ 3.4 MB/s | 900 kB 00:00 2024-09-02T11:52:16.282 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.3 MB/s | 16 MB 00:02 2024-09-02T11:52:16.283 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:16.283 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.9 MB/s | 38 MB 00:04 2024-09-02T11:52:16.319 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-02T11:52:16.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-09-02T11:52:16.352 INFO:teuthology.orchestra.run.smithi094.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 190 kB/s | 80 kB 00:00 2024-09-02T11:52:16.352 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:16.353 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.8 MB/s | 979 kB 00:00 2024-09-02T11:52:16.365 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:16.382 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-02T11:52:16.388 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-09-02T11:52:16.389 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:16.389 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:16.446 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-02T11:52:16.469 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:16.469 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:16.483 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-02T11:52:16.524 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-02T11:52:16.627 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-02T11:52:16.650 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:16.658 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:16.678 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:16.678 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:16.727 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-02T11:52:16.784 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-02T11:52:16.797 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T11:52:16.867 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:16.902 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:28 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:16.963 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:29 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:17.040 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:17.041 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:52:17.042 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:17.042 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 154 k 2024-09-02T11:52:17.042 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:52:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-02T11:52:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-02T11:52:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-02T11:52:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-02T11:52:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-02T11:52:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:17.043 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:17.044 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:17.044 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-09-02T11:52:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:17.045 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 7.0 M 2024-09-02T11:52:17.045 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2024-09-02T11:52:17.045 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:17.099 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:17.101 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:17.101 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-02T11:52:17.101 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:17.101 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:17.101 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 285 k 2024-09-02T11:52:17.101 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:17.102 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:17.102 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:17.102 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:52:17.102 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:17.102 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 285 k 2024-09-02T11:52:17.102 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.3 M 2024-09-02T11:52:17.103 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:17.111 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:17.111 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:17.341 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-volume-19.3.0-4585.gb59673c4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:17.342 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:17.342 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:17.342 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:17.350 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:17.351 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:17.354 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4585.gb59673c4.e 673 kB/s | 154 kB 00:00 2024-09-02T11:52:17.374 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:17.374 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:17.380 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 976 kB/s | 248 kB 00:00 2024-09-02T11:52:17.430 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-02T11:52:17.455 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 766 kB/s | 58 kB 00:00 2024-09-02T11:52:17.481 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-02T11:52:17.503 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-02T11:52:17.535 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:17.535 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/2 2024-09-02T11:52:17.557 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-02T11:52:17.590 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-02T11:52:17.595 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:17.638 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-02T11:52:17.661 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:17.714 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-09-02T11:52:17.716 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:17.716 INFO:teuthology.orchestra.run.smithi038.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-02T11:52:17.734 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:17.750 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:17.751 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:17.751 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-02T11:52:17.751 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:17.789 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:17.800 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:17.801 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:17.805 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T11:52:17.805 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:17.805 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:17.805 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T11:52:17.806 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:17.806 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:17.892 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:17.904 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:17.904 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:17.978 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-02T11:52:18.014 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:18.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-volume 2024-09-02T11:52:18.119 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:18.157 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-02T11:52:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:18.350 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-02T11:52:18.380 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-02T11:52:18.460 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-02T11:52:18.471 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:18.471 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 1/13 2024-09-02T11:52:18.471 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-02T11:52:18.471 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-02T11:52:18.471 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-02T11:52:18.472 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-02T11:52:18.531 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-02T11:52:18.584 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-02T11:52:18.646 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:34 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:18.742 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:18.742 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:18.742 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:18.742 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:18.743 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:18.743 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:18.784 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:18.785 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:18.785 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2024-09-02T11:52:18.785 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:18.785 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:18.785 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 285 k 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 285 k 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 1.3 M 2024-09-02T11:52:18.786 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:18.796 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-02T11:52:18.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:18.796 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:18.796 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:18.796 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-02T11:52:18.797 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-02T11:52:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-02T11:52:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-02T11:52:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-02T11:52:18.798 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:18.798 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:19.016 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-volume-19.3.0-4585.gb59673c4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:19.017 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:19.017 INFO:teuthology.orchestra.run.smithi094.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:19.017 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:19.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-09-02T11:52:19.025 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:19.025 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:19.039 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-cephadm 2024-09-02T11:52:19.047 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:19.047 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:19.140 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-02T11:52:19.151 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-02T11:52:19.230 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-02T11:52:19.266 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:19.273 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-02T11:52:19.307 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-02T11:52:19.326 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:19.336 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:19.342 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:19.342 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:19.342 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-02T11:52:19.342 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:19.590 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:32 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:19.595 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:28 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:19.726 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 126 k 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:19.727 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:19.728 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:52:19.728 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:19.728 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 126 k 2024-09-02T11:52:19.728 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 456 k 2024-09-02T11:52:19.728 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 154 k 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-02T11:52:19.729 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:19.730 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 7.0 M 2024-09-02T11:52:19.731 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 33 M 2024-09-02T11:52:19.731 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:19.931 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.3.0-4585.gb59673c4.el9.x86_64 628 kB/s | 126 kB 00:00 2024-09-02T11:52:19.932 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:19.932 INFO:teuthology.orchestra.run.smithi087.stdout:Total 624 kB/s | 126 kB 00:00 2024-09-02T11:52:19.932 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:19.935 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:19.936 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:20.000 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:20.000 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:20.086 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/6 2024-09-02T11:52:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-02T11:52:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-02T11:52:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-02T11:52:20.087 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-02T11:52:20.150 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:20.250 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:20.257 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-09-02T11:52:20.315 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:20.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:20.315 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:20.315 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:20.315 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:20.315 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:20.316 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-02T11:52:20.341 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4585.gb59673c4.e 720 kB/s | 154 kB 00:00 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-02T11:52:20.351 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-02T11:52:20.352 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-02T11:52:20.352 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-02T11:52:20.352 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:20.352 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:20.368 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-02T11:52:20.409 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-02T11:52:20.574 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:20.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-09-02T11:52:20.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados-devel 2024-09-02T11:52:20.656 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:20.934 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:20.934 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:20.934 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:21.150 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:33 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:21.168 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:37 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:21.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-09-02T11:52:21.288 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:21.288 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 900 k 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-09-02T11:52:21.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:21.290 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 979 k 2024-09-02T11:52:21.290 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.9 M 2024-09-02T11:52:21.290 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:21.307 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:21.307 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:21.307 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 126 k 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:52:21.308 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:21.309 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 126 k 2024-09-02T11:52:21.309 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 456 k 2024-09-02T11:52:21.309 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:21.497 INFO:teuthology.orchestra.run.smithi094.stdout:librados-devel-19.3.0-4585.gb59673c4.el9.x86_64 673 kB/s | 126 kB 00:00 2024-09-02T11:52:21.497 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:21.498 INFO:teuthology.orchestra.run.smithi094.stdout:Total 667 kB/s | 126 kB 00:00 2024-09-02T11:52:21.498 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:21.503 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:21.503 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:21.566 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:21.566 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:21.650 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-02T11:52:21.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-02T11:52:21.654 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:21.654 INFO:teuthology.orchestra.run.smithi099.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-09-02T11:52:21.679 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:21.724 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-02T11:52:21.728 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:21.729 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:21.741 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:21.741 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:21.762 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:34 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:21.825 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:21.840 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:21.841 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:21.855 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:21.866 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-19.3.0-4585.gb59673c4.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-02T11:52:21.866 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:21.867 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.7 MB/s | 979 kB 00:00 2024-09-02T11:52:21.878 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:21.899 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:21.900 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:21.900 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:21.984 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:21.984 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:22.004 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-09-02T11:52:22.164 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:22.177 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:22.233 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T11:52:22.342 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-02T11:52:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:22.402 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-02T11:52:22.481 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:22.481 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:22.481 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:22.481 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:22.481 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:22.481 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:22.558 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:35 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:22.690 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 32 k 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:22.691 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:22.692 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:52:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:22.692 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 k 2024-09-02T11:52:22.692 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 142 k 2024-09-02T11:52:22.692 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:22.694 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs2 2024-09-02T11:52:22.841 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.3.0-4585.gb59673c4.el9.x86_6 213 kB/s | 32 kB 00:00 2024-09-02T11:52:22.841 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:22.841 INFO:teuthology.orchestra.run.smithi087.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-02T11:52:22.842 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:22.845 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:22.845 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:22.849 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:22.852 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:22.931 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-02T11:52:22.942 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:23.022 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-02T11:52:23.062 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/2 2024-09-02T11:52:23.090 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-02T11:52:23.122 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:23.264 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:39 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:23.333 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T11:52:23.334 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:23.334 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:23.334 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T11:52:23.334 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:23.334 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:23.355 INFO:teuthology.orchestra.run.smithi094.stdout:Package libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:23.399 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:23.400 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:23.400 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:23.429 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:23.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install libcephfs-devel 2024-09-02T11:52:23.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-volume 2024-09-02T11:52:23.656 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:23.657 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:23.657 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:23.657 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:23.657 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:23.657 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:23.839 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-09-02T11:52:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/6 2024-09-02T11:52:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-02T11:52:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-02T11:52:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-02T11:52:23.903 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-02T11:52:24.025 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:40 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:24.158 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 32 k 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 32 k 2024-09-02T11:52:24.159 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 142 k 2024-09-02T11:52:24.160 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:24.176 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:24.177 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:24.179 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:36 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 12/12 2024-09-02T11:52:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4585.gb5967 1/12 2024-09-02T11:52:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-02T11:52:24.265 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-02T11:52:24.266 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-02T11:52:24.301 INFO:teuthology.orchestra.run.smithi094.stdout:libcephfs-devel-19.3.0-4585.gb59673c4.el9.x86_6 225 kB/s | 32 kB 00:00 2024-09-02T11:52:24.301 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:24.302 INFO:teuthology.orchestra.run.smithi094.stdout:Total 222 kB/s | 32 kB 00:00 2024-09-02T11:52:24.302 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:24.305 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:24.305 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:24.309 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:24.310 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:24.319 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:24.320 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-02T11:52:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 285 k 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 285 k 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.3 M 2024-09-02T11:52:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:24.395 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:37 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:24.395 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:24.439 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2024-09-02T11:52:24.485 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:24.496 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:24.527 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:24.528 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:24.528 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:24.545 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-volume-19.3.0-4585.gb59673c4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:24.546 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:24.546 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:24.546 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:24.555 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:24.556 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:24.579 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:24.579 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:24.603 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-09-02T11:52:24.663 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-02T11:52:24.664 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:24.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-02T11:52:24.665 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-02T11:52:24.665 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:24.665 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:24.805 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:24.844 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:24.864 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:24.880 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:24.881 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:24.881 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-02T11:52:24.881 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:24.991 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:33 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:25.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-09-02T11:52:25.054 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:25.054 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:25.054 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:25.054 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:25.054 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:25.054 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:25.123 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:25.123 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 900 k 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-09-02T11:52:25.124 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:25.125 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 979 k 2024-09-02T11:52:25.125 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 2.9 M 2024-09-02T11:52:25.125 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:25.155 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:25.240 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librados2 2024-09-02T11:52:25.246 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:25.289 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:25.290 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:25.290 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:25.343 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 663 kB/s | 80 kB 00:00 2024-09-02T11:52:25.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-09-02T11:52:25.510 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-19.3.0-4585.gb59673c4.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-02T11:52:25.511 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:25.511 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.5 MB/s | 979 kB 00:00 2024-09-02T11:52:25.523 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:25.545 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:25.545 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:25.614 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:22 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:25.625 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:25.625 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:25.754 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 49 k 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:52:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-02T11:52:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:25.757 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:25.757 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:25.757 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-09-02T11:52:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:25.757 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-09-02T11:52:25.758 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-09-02T11:52:25.758 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:25.790 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:41 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:25.818 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:25.826 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:25.826 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:25.826 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:25.879 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:25.921 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:25.922 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:25.922 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:25.926 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:25.962 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T11:52:26.000 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install librbd1 2024-09-02T11:52:26.015 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:26.041 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:26.058 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:26.059 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:26.059 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:26.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-09-02T11:52:26.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-09-02T11:52:26.166 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): ceph-mgr-rook-19.3.0-4585.gb59673c4.el9 302 kB/s | 49 kB 00:00 2024-09-02T11:52:26.517 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-02T11:52:26.548 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:42 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:26.558 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-02T11:52:26.636 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:26.668 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:26.668 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/2 2024-09-02T11:52:26.679 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:26.680 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:26.680 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:26.691 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:39 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:26.702 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:38 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:26.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rados 2024-09-02T11:52:26.780 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:26.823 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:26.824 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:26.841 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:26.841 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 126 k 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 126 k 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 456 k 2024-09-02T11:52:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:26.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-09-02T11:52:26.930 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T11:52:26.930 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:26.930 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:26.930 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T11:52:26.930 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:26.930 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:26.944 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 286 kB/s | 222 kB 00:00 2024-09-02T11:52:27.058 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-19.3.0-4585.gb59673c4.el9.x86_64 587 kB/s | 126 kB 00:00 2024-09-02T11:52:27.059 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:27.059 INFO:teuthology.orchestra.run.smithi038.stdout:Total 583 kB/s | 126 kB 00:00 2024-09-02T11:52:27.059 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:27.065 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:27.065 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:27.131 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:27.131 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:27.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-volume 2024-09-02T11:52:27.284 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 207 kB/s | 159 kB 00:00 2024-09-02T11:52:27.296 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:43 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:27.334 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 649 kB/s | 32 kB 00:00 2024-09-02T11:52:27.368 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 345 kB/s | 279 kB 00:00 2024-09-02T11:52:27.386 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:27.393 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 241 kB/s | 14 kB 00:00 2024-09-02T11:52:27.398 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:27.427 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.7 MB/s | 220 kB 00:00 2024-09-02T11:52:27.428 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:27.430 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:27.430 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:27.453 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:40 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:27.461 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-02T11:52:27.494 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-02T11:52:27.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rgw 2024-09-02T11:52:27.528 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-02T11:52:27.542 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:27.584 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:27.585 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:27.585 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:27.594 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 82 kB/s | 54 kB 00:00 2024-09-02T11:52:27.598 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:27.598 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-02T11:52:27.662 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-09-02T11:52:27.669 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:27.686 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:27.686 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:27.777 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:27.778 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:27.785 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:36 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:27.920 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:27.921 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-09-02T11:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 285 k 2024-09-02T11:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:27.922 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 285 k 2024-09-02T11:52:27.923 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.3 M 2024-09-02T11:52:27.924 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:28.052 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:44 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:28.115 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:28.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:28.115 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:28.115 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:28.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:28.115 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:28.142 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:28.156 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-volume-19.3.0-4585.gb59673c4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:28.157 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:28.157 INFO:teuthology.orchestra.run.smithi099.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:28.157 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:28.165 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:28.166 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:28.184 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:28.185 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:28.185 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:28.188 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:28.188 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:28.216 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:28.256 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-cephfs 2024-09-02T11:52:28.302 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:28.305 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:28.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-09-02T11:52:28.348 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:28.349 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:28.349 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:28.411 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:28.443 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-09-02T11:52:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-02T11:52:28.517 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:28.534 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:28.534 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:28.534 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-02T11:52:28.534 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-02T11:52:28.603 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-02T11:52:28.639 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-02T11:52:28.681 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-02T11:52:28.786 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-02T11:52:28.805 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:44 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:28.894 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-02T11:52:28.915 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:40 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:28.937 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:28.938 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:28.938 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:28.957 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-02T11:52:28.991 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:29.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install python3-rbd 2024-09-02T11:52:29.010 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:29.055 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:29.056 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:29.056 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:29.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-09-02T11:52:29.122 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 85 k 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:29.123 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:29.124 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:29.124 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:52:29.124 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:29.124 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2024-09-02T11:52:29.124 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 231 k 2024-09-02T11:52:29.124 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:29.327 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.3.0-4585.gb59673c4.el9.x86_64.rpm 420 kB/s | 85 kB 00:00 2024-09-02T11:52:29.328 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:29.328 INFO:teuthology.orchestra.run.smithi087.stdout:Total 417 kB/s | 85 kB 00:00 2024-09-02T11:52:29.328 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:29.335 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:29.335 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:29.397 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:29.398 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:29.460 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:29.460 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:29.460 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:29.460 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:29.460 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:29.460 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:29.536 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:29.565 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:45 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:29.641 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:29.654 INFO:teuthology.orchestra.run.smithi094.stdout:Package python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:29.681 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:41 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:29.698 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:29.699 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:29.700 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:29.715 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-02T11:52:29.743 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2024-09-02T11:52:29.762 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-fuse 2024-09-02T11:52:29.771 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-02T11:52:29.808 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 32 k 2024-09-02T11:52:29.820 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:29.821 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:29.821 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:29.821 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:52:29.821 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:29.822 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 32 k 2024-09-02T11:52:29.822 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 142 k 2024-09-02T11:52:29.822 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:29.854 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-02T11:52:29.882 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:29.978 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-19.3.0-4585.gb59673c4.el9.x86_6 203 kB/s | 32 kB 00:00 2024-09-02T11:52:29.979 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:29.979 INFO:teuthology.orchestra.run.smithi038.stdout:Total 200 kB/s | 32 kB 00:00 2024-09-02T11:52:29.979 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:29.983 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:29.983 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:29.987 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:29.987 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:30.078 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:30.194 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:30.267 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:30.267 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:30.268 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:30.268 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:30.268 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:30.268 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:30.302 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:39 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:30.306 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:46 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:30.439 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 126 k 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:30.441 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:30.442 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:52:30.442 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:30.442 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 126 k 2024-09-02T11:52:30.442 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 456 k 2024-09-02T11:52:30.442 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:30.443 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:30.443 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:30.443 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:30.443 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:30.443 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:30.443 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 85 k 2024-09-02T11:52:30.444 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:30.444 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:30.444 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:30.444 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:52:30.444 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:30.444 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 85 k 2024-09-02T11:52:30.445 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 231 k 2024-09-02T11:52:30.445 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:30.571 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-09-02T11:52:30.613 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-fuse-19.3.0-4585.gb59673c4.el9.x86_64.rpm 495 kB/s | 85 kB 00:00 2024-09-02T11:52:30.613 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:30.614 INFO:teuthology.orchestra.run.smithi094.stdout:Total 491 kB/s | 85 kB 00:00 2024-09-02T11:52:30.614 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:30.620 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:30.621 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:30.636 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-19.3.0-4585.gb59673c4.el9.x86_64 647 kB/s | 126 kB 00:00 2024-09-02T11:52:30.636 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:30.636 INFO:teuthology.orchestra.run.smithi099.stdout:Total 643 kB/s | 126 kB 00:00 2024-09-02T11:52:30.637 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:30.642 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:30.642 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:30.683 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:30.683 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 13/13 2024-09-02T11:52:30.687 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 1/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-02T11:52:30.688 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-02T11:52:30.705 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:30.706 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:30.773 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:30.773 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:30.833 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:30.857 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:30.930 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-09-02T11:52:30.941 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:30.956 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-02T11:52:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:31.017 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-02T11:52:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:31.019 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:31.142 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:44 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:31.277 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:31.277 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:31.277 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-02T11:52:31.277 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.2 M 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 14 M 2024-09-02T11:52:31.278 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:31.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-09-02T11:52:31.319 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:31.381 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:31.492 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:31.572 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:31.572 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:31.572 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:31.572 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:31.572 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:31.572 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:31.584 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:31.614 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.3.0-4585.gb59673c4.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-02T11:52:31.615 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:31.615 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-02T11:52:31.615 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:31.624 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:31.625 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:31.629 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:31.630 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:31.630 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:31.641 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:31.642 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:31.642 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:31.642 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:31.642 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:31.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-09-02T11:52:31.707 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:31.707 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:31.871 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:28 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:31.877 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-mirror 2024-09-02T11:52:31.886 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2024-09-02T11:52:32.009 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 154 k 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-02T11:52:32.010 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-09-02T11:52:32.011 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-09-02T11:52:32.012 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:32.157 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:32.232 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:32.256 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:32.257 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:32.257 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:32.257 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T11:52:32.257 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:32.257 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:32.257 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:32.348 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:32.394 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:32.395 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:32.395 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:32.445 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:48 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:32.450 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:41 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:32.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-09-02T11:52:32.472 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-02T11:52:32.497 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4585.gb59673c4.e 658 kB/s | 154 kB 00:00 2024-09-02T11:52:32.540 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:32.547 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-02T11:52:32.573 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.3 MB/s | 548 kB 00:00 2024-09-02T11:52:32.584 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:32.584 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repo Size 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:32.585 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 3.2 M 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 14 M 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:32.586 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:32.598 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-02T11:52:32.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2024-09-02T11:52:32.925 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-mirror-19.3.0-4585.gb59673c4.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-02T11:52:32.926 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:32.926 INFO:teuthology.orchestra.run.smithi094.stdout:Total 9.3 MB/s | 3.2 MB 00:00 2024-09-02T11:52:32.926 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:32.935 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:32.935 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:33.017 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:33.017 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:33.029 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:33.121 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:33.167 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:33.168 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:33.168 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:33.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-09-02T11:52:33.235 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:42 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:33.357 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:33.358 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:33.358 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:33.358 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:33.358 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:33.358 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:33.367 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:33.367 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 32 k 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:52:33.368 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:33.369 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 32 k 2024-09-02T11:52:33.369 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 142 k 2024-09-02T11:52:33.369 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:33.462 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:33.522 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-19.3.0-4585.gb59673c4.el9.x86_6 207 kB/s | 32 kB 00:00 2024-09-02T11:52:33.523 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:33.523 INFO:teuthology.orchestra.run.smithi099.stdout:Total 205 kB/s | 32 kB 00:00 2024-09-02T11:52:33.523 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:33.526 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:33.526 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:33.526 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:33.530 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:33.531 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:33.551 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:33.551 INFO:teuthology.orchestra.run.smithi094.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:33.551 INFO:teuthology.orchestra.run.smithi094.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T11:52:33.552 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:33.552 INFO:teuthology.orchestra.run.smithi094.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:33.552 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:33.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-09-02T11:52:33.621 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:33.717 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:33.803 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:33.896 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:33.941 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:33.942 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:33.942 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:34.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-09-02T11:52:34.062 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:34.186 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:47 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:34.283 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:34.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:34.284 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:34.284 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:34.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:34.284 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 170 k 2024-09-02T11:52:34.321 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 170 k 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 490 k 2024-09-02T11:52:34.322 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-02T11:52:34.443 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2024-09-02T11:52:34.535 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.3.0-4585.gb59673c4.el9.x86_64.rpm 797 kB/s | 170 kB 00:00 2024-09-02T11:52:34.536 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:34.536 INFO:teuthology.orchestra.run.smithi087.stdout:Total 792 kB/s | 170 kB 00:00 2024-09-02T11:52:34.536 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-02T11:52:34.543 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-02T11:52:34.543 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-02T11:52:34.570 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:34.588 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:34.588 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:34.589 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:34.589 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:34.589 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:34.589 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:34.607 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-02T11:52:34.607 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-02T11:52:34.663 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:34.709 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:34.710 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:34.710 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:34.769 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-02T11:52:34.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-09-02T11:52:34.790 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-09-02T11:52:34.793 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:34.793 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.5 MB/s | 7.0 MB 00:02 2024-09-02T11:52:34.855 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install rbd-nbd 2024-09-02T11:52:34.867 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:34.879 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:34.879 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:34.889 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:34.982 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:34.982 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:34.991 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:43 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:35.079 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:35.122 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:35.123 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:35.123 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:35.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2024-09-02T11:52:35.297 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:35.343 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:47 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:35.416 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:51 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:35.435 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:35.481 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:35.482 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:35.482 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:35.544 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:35.544 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:35.545 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-02T11:52:35.545 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:35.545 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:52:35.545 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:35.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-09-02T11:52:35.551 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 170 k 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2024-09-02T11:52:35.552 INFO:teuthology.orchestra.run.smithi094.stdout:================================================================================ 2024-09-02T11:52:35.553 INFO:teuthology.orchestra.run.smithi094.stdout:Install 1 Package 2024-09-02T11:52:35.553 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:35.553 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 170 k 2024-09-02T11:52:35.553 INFO:teuthology.orchestra.run.smithi094.stdout:Installed size: 490 k 2024-09-02T11:52:35.553 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2024-09-02T11:52:35.588 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-02T11:52:35.635 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-02T11:52:35.745 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:44 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:35.771 INFO:teuthology.orchestra.run.smithi094.stdout:rbd-nbd-19.3.0-4585.gb59673c4.el9.x86_64.rpm 783 kB/s | 170 kB 00:00 2024-09-02T11:52:35.772 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:35.772 INFO:teuthology.orchestra.run.smithi094.stdout:Total 777 kB/s | 170 kB 00:00 2024-09-02T11:52:35.772 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2024-09-02T11:52:35.780 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2024-09-02T11:52:35.780 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2024-09-02T11:52:35.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install nvme-cli 2024-09-02T11:52:35.835 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:35.843 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2024-09-02T11:52:35.844 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2024-09-02T11:52:35.878 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:35.879 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:35.879 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:35.953 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2024-09-02T11:52:35.992 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2024-09-02T11:52:36.100 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:36.105 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-02T11:52:36.244 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:36.244 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 85 k 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-09-02T11:52:36.245 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 231 k 2024-09-02T11:52:36.246 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-02T11:52:36.348 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-02T11:52:36.366 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:49 ago on Mon 02 Sep 2024 11:50:47 AM UTC. 2024-09-02T11:52:36.379 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:36.439 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-19.3.0-4585.gb59673c4.el9.x86_64.rpm 440 kB/s | 85 kB 00:00 2024-09-02T11:52:36.440 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:36.440 INFO:teuthology.orchestra.run.smithi038.stdout:Total 437 kB/s | 85 kB 00:00 2024-09-02T11:52:36.440 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:36.447 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:36.448 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:36.456 INFO:teuthology.orchestra.run.smithi087.stdout:Package nvme-cli-2.9.1-6.el9.x86_64 is already installed. 2024-09-02T11:52:36.500 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-02T11:52:36.501 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-02T11:52:36.502 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-02T11:52:36.510 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:45 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:36.513 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:36.513 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:36.546 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:36.587 DEBUG:teuthology.parallel:result is None 2024-09-02T11:52:36.599 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:36.642 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:36.643 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:36.643 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:36.659 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:36.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2024-09-02T11:52:36.755 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:36.788 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:36.788 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:36.789 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2024-09-02T11:52:36.789 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:36.789 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:52:36.789 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:37.039 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install nvme-cli 2024-09-02T11:52:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:37.109 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 6/6 2024-09-02T11:52:37.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 1/6 2024-09-02T11:52:37.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-02T11:52:37.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-02T11:52:37.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-02T11:52:37.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-02T11:52:37.279 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:46 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:37.331 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:37.332 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-02T11:52:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-02T11:52:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-02T11:52:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-02T11:52:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:37.367 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:37.370 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:37.413 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:37.415 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:37.415 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:37.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2024-09-02T11:52:37.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-09-02T11:52:37.597 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:53 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T11:52:37.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-09-02T11:52:37.687 INFO:teuthology.orchestra.run.smithi094.stdout:Package nvme-cli-2.9.1-6.el9.x86_64 is already installed. 2024-09-02T11:52:37.732 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2024-09-02T11:52:37.733 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2024-09-02T11:52:37.733 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2024-09-02T11:52:37.801 DEBUG:teuthology.parallel:result is None 2024-09-02T11:52:38.045 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:47 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:38.134 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:38.165 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:50 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:38.176 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:38.177 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:38.177 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:38.231 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:35 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:38.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2024-09-02T11:52:38.303 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:38.304 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-02T11:52:38.304 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.2 M 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 14 M 2024-09-02T11:52:38.305 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:38.369 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:38.369 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:38.369 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:52:38.369 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 900 k 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-09-02T11:52:38.370 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:38.371 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 979 k 2024-09-02T11:52:38.371 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.9 M 2024-09-02T11:52:38.371 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:38.508 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 684 kB/s | 80 kB 00:00 2024-09-02T11:52:38.626 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-19.3.0-4585.gb59673c4.el9.x86_64.rpm 9.9 MB/s | 3.2 MB 00:00 2024-09-02T11:52:38.626 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:38.627 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.8 MB/s | 3.2 MB 00:00 2024-09-02T11:52:38.627 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:38.636 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:38.637 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:38.667 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.3.0-4585.gb59673c4.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-02T11:52:38.667 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:38.668 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.2 MB/s | 979 kB 00:00 2024-09-02T11:52:38.679 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:38.701 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:38.701 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:38.722 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:38.723 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:38.784 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:38.784 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:38.811 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:47 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:38.900 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:38.944 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:38.945 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:38.945 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:38.982 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:39.020 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2024-09-02T11:52:39.123 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T11:52:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:39.279 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:39.279 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T11:52:39.279 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:39.279 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:39.570 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:48 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:39.704 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:39.705 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:39.705 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:39.705 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:39.705 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:39.706 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 85 k 2024-09-02T11:52:39.706 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:39.706 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:39.706 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:39.706 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:52:39.707 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:39.707 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 85 k 2024-09-02T11:52:39.707 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 231 k 2024-09-02T11:52:39.707 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2/2 2024-09-02T11:52:39.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/2 2024-09-02T11:52:39.900 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-19.3.0-4585.gb59673c4.el9.x86_64.rpm 437 kB/s | 85 kB 00:00 2024-09-02T11:52:39.900 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:39.900 INFO:teuthology.orchestra.run.smithi099.stdout:Total 434 kB/s | 85 kB 00:00 2024-09-02T11:52:39.901 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:39.907 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:39.908 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:39.971 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:39.971 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:40.118 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:40.157 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T11:52:40.158 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:40.158 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:40.158 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T11:52:40.158 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:40.158 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:40.220 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:40.332 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:40.332 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:40.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-09-02T11:52:40.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-09-02T11:52:40.576 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:40.885 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:40.885 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:40.885 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:40.885 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:40.885 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:40.885 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:41.044 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:41.135 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:53 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:41.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2024-09-02T11:52:41.184 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.3.0-4585.gb59673c4.el9 ceph-noarch 285 k 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:41.185 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:41.186 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:41.186 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:52:41.186 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:41.186 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 285 k 2024-09-02T11:52:41.186 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.3 M 2024-09-02T11:52:41.186 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:41.275 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:41.275 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:41.275 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 170 k 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 170 k 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 490 k 2024-09-02T11:52:41.276 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-02T11:52:41.413 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-volume-19.3.0-4585.gb59673c4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:41.414 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:41.414 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-02T11:52:41.414 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:41.423 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:41.423 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:41.445 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:41.446 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:41.489 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-19.3.0-4585.gb59673c4.el9.x86_64.rpm 806 kB/s | 170 kB 00:00 2024-09-02T11:52:41.489 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:41.489 INFO:teuthology.orchestra.run.smithi038.stdout:Total 800 kB/s | 170 kB 00:00 2024-09-02T11:52:41.489 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-02T11:52:41.497 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-02T11:52:41.497 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-02T11:52:41.565 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-02T11:52:41.566 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-02T11:52:41.669 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:41.726 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-02T11:52:41.748 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:41.752 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:50 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:41.764 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:41.764 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:41.764 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-02T11:52:41.764 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:41.828 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:41.889 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:41.891 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:41.891 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:52:41.891 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:41.891 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.2 M 2024-09-02T11:52:41.891 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 14 M 2024-09-02T11:52:41.891 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:42.192 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:42.220 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-19.3.0-4585.gb59673c4.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-02T11:52:42.221 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:42.221 INFO:teuthology.orchestra.run.smithi099.stdout:Total 9.6 MB/s | 3.2 MB 00:00 2024-09-02T11:52:42.221 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:42.230 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:42.230 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:42.312 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:42.312 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:42.439 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:42.439 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:42.439 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-02T11:52:42.440 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:42.440 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T11:52:42.440 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:42.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install nvme-cli 2024-09-02T11:52:42.736 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 1/1 2024-09-02T11:52:42.736 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:42.736 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:42.736 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.3.0-4585.gb59673c4.el9.noarch 2024-09-02T11:52:42.736 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:42.736 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:42.753 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:42.829 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:42.855 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:42.856 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:42.856 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T11:52:42.856 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:42.856 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:42.856 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:43.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-09-02T11:52:43.220 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:55 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T11:52:43.312 INFO:teuthology.orchestra.run.smithi038.stdout:Package nvme-cli-2.9.1-6.el9.x86_64 is already installed. 2024-09-02T11:52:43.358 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-02T11:52:43.359 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-02T11:52:43.359 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-02T11:52:43.433 DEBUG:teuthology.parallel:result is None 2024-09-02T11:52:43.611 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:40 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:43.750 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 126 k 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:43.751 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:43.752 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:43.752 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:52:43.752 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:43.752 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 126 k 2024-09-02T11:52:43.752 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-09-02T11:52:43.752 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:43.945 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.3.0-4585.gb59673c4.el9.x86_64 662 kB/s | 126 kB 00:00 2024-09-02T11:52:43.945 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:43.945 INFO:teuthology.orchestra.run.smithi064.stdout:Total 657 kB/s | 126 kB 00:00 2024-09-02T11:52:43.946 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:43.950 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:43.950 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:43.965 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:43.965 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:43.965 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:43.965 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:43.965 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:43.965 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:44.016 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:44.016 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:44.172 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:44.215 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2024-09-02T11:52:44.268 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:44.710 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:44.781 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:53 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:44.917 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 170 k 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:44.918 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 170 k 2024-09-02T11:52:44.919 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 490 k 2024-09-02T11:52:44.919 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-02T11:52:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:44.970 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:44.970 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:44.971 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:45.124 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-19.3.0-4585.gb59673c4.el9.x86_64.rpm 831 kB/s | 170 kB 00:00 2024-09-02T11:52:45.124 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:45.125 INFO:teuthology.orchestra.run.smithi099.stdout:Total 825 kB/s | 170 kB 00:00 2024-09-02T11:52:45.125 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-02T11:52:45.132 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-02T11:52:45.132 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-02T11:52:45.197 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-02T11:52:45.197 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-02T11:52:45.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-09-02T11:52:45.353 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-02T11:52:45.461 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:45.812 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:42 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:45.895 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:45.908 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:45.953 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:45.954 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:45.955 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:46.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-09-02T11:52:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:46.124 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:46.125 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-02T11:52:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:46.125 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T11:52:46.125 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:46.358 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install nvme-cli 2024-09-02T11:52:46.597 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:46.735 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 32 k 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:46.736 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:46.737 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:46.737 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:52:46.737 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:46.737 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 32 k 2024-09-02T11:52:46.737 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 142 k 2024-09-02T11:52:46.737 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:46.888 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.3.0-4585.gb59673c4.el9.x86_6 209 kB/s | 32 kB 00:00 2024-09-02T11:52:46.889 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:46.889 INFO:teuthology.orchestra.run.smithi064.stdout:Total 207 kB/s | 32 kB 00:00 2024-09-02T11:52:46.889 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:46.893 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:46.893 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:46.897 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:46.898 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:46.915 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:55 ago on Mon 02 Sep 2024 11:50:51 AM UTC. 2024-09-02T11:52:46.976 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:47.005 INFO:teuthology.orchestra.run.smithi099.stdout:Package nvme-cli-2.9.1-6.el9.x86_64 is already installed. 2024-09-02T11:52:47.048 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-02T11:52:47.049 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-09-02T11:52:47.049 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-02T11:52:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:47.129 DEBUG:teuthology.parallel:result is None 2024-09-02T11:52:47.407 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:47.631 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:47.631 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:47.631 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:47.631 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:47.631 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:47.631 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:47.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-09-02T11:52:48.380 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:48.475 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:48.520 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:48.521 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:48.521 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:48.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-09-02T11:52:49.165 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:46 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:49.258 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:49.303 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:49.304 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:49.304 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:49.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-09-02T11:52:49.951 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:46 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:50.043 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:50.089 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:50.090 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:50.090 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:50.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-09-02T11:52:50.744 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:50.836 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:50.882 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:50.883 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:50.883 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:50.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-09-02T11:52:51.536 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:48 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:51.629 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:51.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-09-02T11:52:52.336 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:49 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:52.430 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.3.0-4585.gb59673c4.el9.x86_64 is already installed. 2024-09-02T11:52:52.475 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:52.476 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:52:52.476 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:52.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-09-02T11:52:53.134 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:53.273 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:53.274 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:53.274 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:53.274 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:53.274 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:53.274 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 85 k 2024-09-02T11:52:53.274 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-09-02T11:52:53.275 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:53.476 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.3.0-4585.gb59673c4.el9.x86_64.rpm 424 kB/s | 85 kB 00:00 2024-09-02T11:52:53.477 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:53.477 INFO:teuthology.orchestra.run.smithi064.stdout:Total 421 kB/s | 85 kB 00:00 2024-09-02T11:52:53.477 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:53.484 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:53.485 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:53.550 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:53.550 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:53.699 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:53.807 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:54.234 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:54.467 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:54.468 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:54.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-09-02T11:52:55.345 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:52 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:55.484 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 3.2 M 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.2 M 2024-09-02T11:52:55.485 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 14 M 2024-09-02T11:52:55.486 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:55.830 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.3.0-4585.gb59673c4.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-02T11:52:55.830 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:55.831 INFO:teuthology.orchestra.run.smithi064.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-02T11:52:55.831 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:55.840 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:55.841 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:55.925 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:55.926 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:56.386 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:56.455 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:56.480 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:56.481 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T11:52:56.481 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T11:52:56.481 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:56.481 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T11:52:56.481 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:57.600 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:57.600 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:52:57.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-09-02T11:52:58.406 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:55 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:52:58.544 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:52:58.544 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:58.544 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.3.0-4585.gb59673c4.el9 ceph 170 k 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 170 k 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 490 k 2024-09-02T11:52:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-02T11:52:58.759 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.3.0-4585.gb59673c4.el9.x86_64.rpm 799 kB/s | 170 kB 00:00 2024-09-02T11:52:58.759 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:52:58.760 INFO:teuthology.orchestra.run.smithi064.stdout:Total 793 kB/s | 170 kB 00:00 2024-09-02T11:52:58.760 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-02T11:52:58.768 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-02T11:52:58.768 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-02T11:52:58.834 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-02T11:52:58.834 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-02T11:52:58.991 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-02T11:52:59.091 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:59.514 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 1/1 2024-09-02T11:52:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:59.767 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-02T11:52:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.3.0-4585.gb59673c4.el9.x86_64 2024-09-02T11:52:59.767 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:52:59.768 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:53:00.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install nvme-cli 2024-09-02T11:53:00.596 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:57 ago on Mon 02 Sep 2024 11:51:03 AM UTC. 2024-09-02T11:53:00.689 INFO:teuthology.orchestra.run.smithi064.stdout:Package nvme-cli-2.9.1-6.el9.x86_64 is already installed. 2024-09-02T11:53:00.735 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-02T11:53:00.736 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-02T11:53:00.736 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-02T11:53:00.817 DEBUG:teuthology.parallel:result is None 2024-09-02T11:53:00.818 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:00.969 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T11:53:00.994 INFO:teuthology.orchestra.run.smithi038.stdout:19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:00.995 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:00.995 INFO:teuthology.task.install:The correct ceph version 19.3.0-4585.gb59673c4 is installed. 2024-09-02T11:53:00.997 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:01.141 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T11:53:01.163 INFO:teuthology.orchestra.run.smithi064.stdout:19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.163 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.163 INFO:teuthology.task.install:The correct ceph version 19.3.0-4585.gb59673c4 is installed. 2024-09-02T11:53:01.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:01.309 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T11:53:01.334 INFO:teuthology.orchestra.run.smithi087.stdout:19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.334 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.335 INFO:teuthology.task.install:The correct ceph version 19.3.0-4585.gb59673c4 is installed. 2024-09-02T11:53:01.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:01.484 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T11:53:01.509 INFO:teuthology.orchestra.run.smithi094.stdout:19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.509 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.509 INFO:teuthology.task.install:The correct ceph version 19.3.0-4585.gb59673c4 is installed. 2024-09-02T11:53:01.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:01.656 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T11:53:01.682 INFO:teuthology.orchestra.run.smithi099.stdout:19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.682 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4585.gb59673c4.el9 2024-09-02T11:53:01.682 INFO:teuthology.task.install:The correct ceph version 19.3.0-4585.gb59673c4 is installed. 2024-09-02T11:53:01.685 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-02T11:53:01.685 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:53:01.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T11:53:01.716 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:53:01.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T11:53:01.742 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:53:01.742 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T11:53:01.774 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:53:01.774 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T11:53:01.806 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:53:01.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T11:53:01.836 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-02T11:53:01.837 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:53:01.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T11:53:01.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T11:53:01.937 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:53:01.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T11:53:01.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T11:53:02.031 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:53:02.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T11:53:02.062 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T11:53:02.131 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:53:02.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T11:53:02.160 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T11:53:02.226 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:53:02.226 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T11:53:02.256 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T11:53:02.325 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-02T11:53:02.325 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:53:02.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T11:53:02.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T11:53:02.424 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:53:02.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T11:53:02.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T11:53:02.517 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:53:02.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T11:53:02.543 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T11:53:02.608 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:53:02.608 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T11:53:02.634 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T11:53:02.698 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:53:02.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T11:53:02.728 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T11:53:02.798 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-02T11:53:02.799 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:53:02.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T11:53:02.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T11:53:02.899 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:53:02.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T11:53:02.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T11:53:02.993 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:53:02.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T11:53:03.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T11:53:03.082 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:53:03.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T11:53:03.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T11:53:03.173 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:53:03.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T11:53:03.204 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T11:53:03.273 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-02T11:53:03.373 INFO:tasks.cephadm:Config: {'watchdog_setup': None, 'conf': {'global': {'mon warn on pool no app': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_FAILED_DAEMON', 'is in error state', 'failed cephadm daemon'], 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf'} 2024-09-02T11:53:03.374 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:03.374 INFO:tasks.cephadm:Cluster fsid is e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:53:03.374 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-02T11:53:03.375 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.64', 'mon.b': '172.21.15.87', 'mon.c': '172.21.15.94'} 2024-09-02T11:53:03.375 INFO:tasks.cephadm:First mon is mon.a on smithi064 2024-09-02T11:53:03.375 INFO:tasks.cephadm:First mgr is x 2024-09-02T11:53:03.375 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-02T11:53:03.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-09-02T11:53:03.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-09-02T11:53:03.434 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname $(hostname -s) 2024-09-02T11:53:03.460 DEBUG:teuthology.orchestra.run.smithi094:> sudo hostname $(hostname -s) 2024-09-02T11:53:03.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname $(hostname -s) 2024-09-02T11:53:03.515 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-02T11:53:03.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:03.664 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b59673c44bd569f9f3db37f87bced695dec5fcbf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82619/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4585-gb59673c4', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4585.gb59673c4'}, 'url': 'https://4.chacra.ceph.com/r/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-29 09:05:55.668247', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main-nvmeof', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-02T11:53:03.814 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref main-nvmeof, sha1 b59673c44bd569f9f3db37f87bced695dec5fcbf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:53:03.817 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm 2024-09-02T11:53:03.817 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm 2024-09-02T11:53:03.817 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:04.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 802682 Sep 2 11:53 /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:04.401 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:04.977 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 802682 Sep 2 11:53 /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:04.977 DEBUG:teuthology.orchestra.run.smithi087:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:05.657 INFO:teuthology.orchestra.run.smithi087.stdout:-rw-r--r--. 1 ubuntu ubuntu 802682 Sep 2 11:53 /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:05.657 DEBUG:teuthology.orchestra.run.smithi094:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.249 INFO:teuthology.orchestra.run.smithi094.stdout:-rw-r--r--. 1 ubuntu ubuntu 802682 Sep 2 11:53 /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.250 DEBUG:teuthology.orchestra.run.smithi099:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/main-nvmeof/b59673c44bd569f9f3db37f87bced695dec5fcbf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.836 INFO:teuthology.orchestra.run.smithi099.stdout:-rw-r--r--. 1 ubuntu ubuntu 802682 Sep 2 11:53 /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.837 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.856 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.876 DEBUG:teuthology.orchestra.run.smithi087:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.895 DEBUG:teuthology.orchestra.run.smithi094:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.915 DEBUG:teuthology.orchestra.run.smithi099:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T11:53:06.948 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf on all hosts... 2024-09-02T11:53:06.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf pull 2024-09-02T11:53:06.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf pull 2024-09-02T11:53:06.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf pull 2024-09-02T11:53:06.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf pull 2024-09-02T11:53:06.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf pull 2024-09-02T11:53:08.415 INFO:teuthology.orchestra.run.smithi087.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:53:08.484 INFO:teuthology.orchestra.run.smithi099.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:53:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:53:08.565 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:53:08.611 INFO:teuthology.orchestra.run.smithi094.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:53:39.465 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-02T11:53:39.465 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev)", 2024-09-02T11:53:39.465 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238", 2024-09-02T11:53:39.465 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-09-02T11:53:39.465 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81" 2024-09-02T11:53:39.466 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-02T11:53:39.466 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-02T11:53:40.335 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-02T11:53:40.336 INFO:teuthology.orchestra.run.smithi087.stdout: "ceph_version": "ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev)", 2024-09-02T11:53:40.336 INFO:teuthology.orchestra.run.smithi087.stdout: "image_id": "b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238", 2024-09-02T11:53:40.336 INFO:teuthology.orchestra.run.smithi087.stdout: "repo_digests": [ 2024-09-02T11:53:40.336 INFO:teuthology.orchestra.run.smithi087.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81" 2024-09-02T11:53:40.336 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-09-02T11:53:40.336 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-02T11:53:42.225 INFO:teuthology.orchestra.run.smithi094.stdout:{ 2024-09-02T11:53:42.225 INFO:teuthology.orchestra.run.smithi094.stdout: "ceph_version": "ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev)", 2024-09-02T11:53:42.225 INFO:teuthology.orchestra.run.smithi094.stdout: "image_id": "b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238", 2024-09-02T11:53:42.225 INFO:teuthology.orchestra.run.smithi094.stdout: "repo_digests": [ 2024-09-02T11:53:42.225 INFO:teuthology.orchestra.run.smithi094.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81" 2024-09-02T11:53:42.226 INFO:teuthology.orchestra.run.smithi094.stdout: ] 2024-09-02T11:53:42.226 INFO:teuthology.orchestra.run.smithi094.stdout:} 2024-09-02T11:53:54.233 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-02T11:53:54.233 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev)", 2024-09-02T11:53:54.233 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238", 2024-09-02T11:53:54.233 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-09-02T11:53:54.234 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81" 2024-09-02T11:53:54.234 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-02T11:53:54.234 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-02T11:54:08.695 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-09-02T11:54:08.695 INFO:teuthology.orchestra.run.smithi099.stdout: "ceph_version": "ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev)", 2024-09-02T11:54:08.696 INFO:teuthology.orchestra.run.smithi099.stdout: "image_id": "b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238", 2024-09-02T11:54:08.696 INFO:teuthology.orchestra.run.smithi099.stdout: "repo_digests": [ 2024-09-02T11:54:08.696 INFO:teuthology.orchestra.run.smithi099.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81" 2024-09-02T11:54:08.696 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-09-02T11:54:08.696 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-09-02T11:54:08.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-09-02T11:54:08.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-09-02T11:54:08.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph 2024-09-02T11:54:08.820 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph 2024-09-02T11:54:08.854 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph 2024-09-02T11:54:08.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-09-02T11:54:08.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-09-02T11:54:08.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /etc/ceph 2024-09-02T11:54:08.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /etc/ceph 2024-09-02T11:54:08.998 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /etc/ceph 2024-09-02T11:54:09.023 INFO:tasks.cephadm:Writing seed config... 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [global] mon warn on pool no app = False 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-02T11:54:09.024 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-02T11:54:09.025 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:54:09.025 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-02T11:54:09.044 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e8e22764-6921-11ef-bcd4-c7b262605968 mon warn on pool no app = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-02T11:54:09.044 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service 2024-09-02T11:54:09.087 DEBUG:teuthology.orchestra.run.smithi064:mgr.x> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x.service 2024-09-02T11:54:09.130 INFO:tasks.cephadm:Bootstrapping... 2024-09-02T11:54:09.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf -v bootstrap --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.64 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-02T11:54:09.342 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-02T11:54:09.342 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf', '-v', 'bootstrap', '--fsid', 'e8e22764-6921-11ef-bcd4-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.64', '--skip-admin-label'] 2024-09-02T11:54:09.342 INFO:teuthology.orchestra.run.smithi064.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-02T11:54:09.343 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying podman|docker is present... 2024-09-02T11:54:09.380 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 5.2.2 2024-09-02T11:54:09.380 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying lvm2 is present... 2024-09-02T11:54:09.380 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying time synchronization is in place... 2024-09-02T11:54:09.388 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-02T11:54:09.388 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-02T11:54:09.394 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-02T11:54:09.394 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-09-02T11:54:09.401 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-09-02T11:54:09.407 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-09-02T11:54:09.407 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-09-02T11:54:09.407 INFO:teuthology.orchestra.run.smithi064.stdout:Repeating the final host check... 2024-09-02T11:54:09.429 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 5.2.2 2024-09-02T11:54:09.429 INFO:teuthology.orchestra.run.smithi064.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-02T11:54:09.430 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl is present 2024-09-02T11:54:09.430 INFO:teuthology.orchestra.run.smithi064.stdout:lvcreate is present 2024-09-02T11:54:09.436 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-02T11:54:09.436 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-02T11:54:09.442 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-02T11:54:09.442 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-09-02T11:54:09.449 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Host looks OK 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Cluster fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Acquiring lock 140042246599824 on /run/cephadm/e8e22764-6921-11ef-bcd4-c7b262605968.lock 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Lock 140042246599824 acquired on /run/cephadm/e8e22764-6921-11ef-bcd4-c7b262605968.lock 2024-09-02T11:54:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 3300 ... 2024-09-02T11:54:09.457 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 6789 ... 2024-09-02T11:54:09.457 INFO:teuthology.orchestra.run.smithi064.stdout:Base mon IP(s) is [172.21.15.64:3300, 172.21.15.64:6789], mon addrv is [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-09-02T11:54:09.460 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.64 metric 100 2024-09-02T11:54:09.460 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2024-09-02T11:54:09.462 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-02T11:54:09.462 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-02T11:54:09.462 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-02T11:54:09.465 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-02T11:54:09.465 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-02T11:54:09.465 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-02T11:54:09.465 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-02T11:54:09.465 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8df/64 scope link noprefixroute 2024-09-02T11:54:09.465 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-02T11:54:09.466 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-09-02T11:54:09.466 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-09-02T11:54:09.466 INFO:teuthology.orchestra.run.smithi064.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-02T11:54:09.466 INFO:teuthology.orchestra.run.smithi064.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-02T11:54:09.466 INFO:teuthology.orchestra.run.smithi064.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:54:09.921 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 2024-09-02T11:54:09.921 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf... 2024-09-02T11:54:09.921 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Getting image source signatures 2024-09-02T11:54:09.921 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:630c1694669dd8aeed84a4a7b5b970eda4a7f40d7529ba87d203ae0993a641c5 2024-09-02T11:54:09.922 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-02T11:54:09.922 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying config sha256:b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 2024-09-02T11:54:09.922 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-02T11:54:10.818 INFO:teuthology.orchestra.run.smithi064.stdout:ceph: stdout ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev) 2024-09-02T11:54:10.818 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph version: ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev) 2024-09-02T11:54:10.818 INFO:teuthology.orchestra.run.smithi064.stdout:Extracting ceph user uid/gid from container image... 2024-09-02T11:54:11.807 INFO:teuthology.orchestra.run.smithi064.stdout:stat: stdout 167 167 2024-09-02T11:54:11.807 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial keys... 2024-09-02T11:54:12.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQBkp9VmCSJgBxAAO6vF+oUpPMXxxA74Gi6iUA== 2024-09-02T11:54:13.586 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQBlp9VmdMHZAhAAxuLyGm2NiLkCnu5TTblavQ== 2024-09-02T11:54:14.516 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQBlp9Vm9DVQNxAA5mp71rOwybZyzHLEykHjhQ== 2024-09-02T11:54:14.517 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial monmap... 2024-09-02T11:54:15.607 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-02T11:54:15.607 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-02T11:54:15.607 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:15.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-02T11:54:15.608 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool for a [v2:172.21.15.64:3300,v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-02T11:54:15.608 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-09-02T11:54:15.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: set fsid to e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:15.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-02T11:54:15.609 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:54:15.609 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mon... 2024-09-02T11:54:16.612 INFO:teuthology.orchestra.run.smithi064.stdout:create mon.a on 2024-09-02T11:54:16.808 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-02T11:54:16.948 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-02T11:54:17.121 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e8e22764-6921-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968.target. 2024-09-02T11:54:17.121 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e8e22764-6921-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968.target. 2024-09-02T11:54:17.366 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T11:54:17.366 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service: Unit ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service not loaded. 2024-09-02T11:54:17.539 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968.target.wants/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service → /etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service. 2024-09-02T11:54:18.086 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T11:54:18.086 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-09-02T11:54:18.093 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T11:54:18.093 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-09-02T11:54:18.093 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-09-02T11:54:18.093 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-09-02T11:54:18.094 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon to start... 2024-09-02T11:54:18.094 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon... 2024-09-02T11:54:18.217 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:18 smithi064 systemd[1]: Started Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T11:54:19.133 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout cluster: 2024-09-02T11:54:19.133 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout id: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:19.133 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-02T11:54:19.133 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:19.133 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout services: 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.561287s) 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout data: 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pgs: 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:mon is available 2024-09-02T11:54:19.134 INFO:teuthology.orchestra.run.smithi064.stdout:Assimilating anything we can from ceph.conf... 2024-09-02T11:54:19.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-02T11:54:19.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: monmap epoch 1 2024-09-02T11:54:19.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:19.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: last_changed 2024-09-02T11:54:14.838508+0000 2024-09-02T11:54:19.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:54:19.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: min_mon_release 19 (squid) 2024-09-02T11:54:19.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: election_strategy: 1 2024-09-02T11:54:19.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:54:19.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: fsmap 2024-09-02T11:54:19.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: osdmap e1: 0 total, 0 up, 0 in 2024-09-02T11:54:19.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: mgrmap e1: no daemons active 2024-09-02T11:54:19.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:19 smithi064 ceph-mon[16801]: from='client.? 172.21.15.64:0/2332341724' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T11:54:20.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:20 smithi064 ceph-mon[16801]: from='client.? 172.21.15.64:0/1420124552' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-02T11:54:20.235 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:20 smithi064 ceph-mon[16801]: from='client.? 172.21.15.64:0/1420124552' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-02T11:54:20.238 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-02T11:54:20.239 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:20.240 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-09-02T11:54:20.240 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-02T11:54:20.240 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-02T11:54:20.240 INFO:teuthology.orchestra.run.smithi064.stdout:Generating new minimal ceph.conf... 2024-09-02T11:54:21.219 INFO:teuthology.orchestra.run.smithi064.stdout:Restarting the monitor... 2024-09-02T11:54:21.239 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:21 smithi064 ceph-mon[16801]: from='client.? 172.21.15.64:0/1622168824' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:54:21.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:21 smithi064 systemd[1]: Stopping Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T11:54:21.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a[16797]: 2024-09-02T11:54:21.484+0000 7f5e911e3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T11:54:21.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a[16797]: 2024-09-02T11:54:21.484+0000 7f5e911e3640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-02T11:54:21.994 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:21 smithi064 podman[17041]: 2024-09-02 11:54:21.73054834 +0000 UTC m=+0.346634515 container died 80baa12d76a171a25c13de5580729606af6480d0a5d56da715095b48ba49870a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-02T11:54:22.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 podman[17041]: 2024-09-02 11:54:22.093008034 +0000 UTC m=+0.709094251 container cleanup 80baa12d76a171a25c13de5580729606af6480d0a5d56da715095b48ba49870a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c) 2024-09-02T11:54:22.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 bash[17041]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a 2024-09-02T11:54:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 podman[17052]: 2024-09-02 11:54:22.335509076 +0000 UTC m=+0.604898015 container remove 80baa12d76a171a25c13de5580729606af6480d0a5d56da715095b48ba49870a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-02T11:54:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-02T11:54:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 systemd[1]: Stopped Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T11:54:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 systemd[1]: Starting Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T11:54:22.921 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 podman[17145]: 2024-09-02 11:54:22.571780152 +0000 UTC m=+0.015557782 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf 2024-09-02T11:54:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 podman[17145]: 2024-09-02 11:54:22.686348757 +0000 UTC m=+0.130126377 container create 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, io.buildah.version=1.37.2) 2024-09-02T11:54:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 podman[17145]: 2024-09-02 11:54:22.896650997 +0000 UTC m=+0.340428618 container init 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=main-nvmeof-b59673c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-02T11:54:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 podman[17145]: 2024-09-02 11:54:22.899734383 +0000 UTC m=+0.343512004 container start 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-nvmeof-b59673c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T11:54:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: set uid:gid to 167:167 (ceph:ceph) 2024-09-02T11:54:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev), process ceph-mon, pid 2 2024-09-02T11:54:22.922 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: pidfile_write: ignore empty --pid-file 2024-09-02T11:54:23.001 INFO:teuthology.orchestra.run.smithi064.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-02T11:54:23.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: load: jerasure load: lrc 2024-09-02T11:54:23.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: RocksDB version: 7.9.2 2024-09-02T11:54:23.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Git sha 0 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Compile date 2024-08-29 08:01:36 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: DB SUMMARY 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: DB Session ID: R3TX8JTIWITAFMIMAQGC 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: CURRENT file: CURRENT 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: IDENTITY file: IDENTITY 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89256 ; 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.error_if_exists: 0 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.create_if_missing: 0 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.paranoid_checks: 1 2024-09-02T11:54:23.250 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.env: 0x5577b6706160 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.fs: PosixFileSystem 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.info_log: 0x5577b702f880 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_file_opening_threads: 16 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.statistics: (nil) 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.use_fsync: 0 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_log_file_size: 0 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-02T11:54:23.251 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.keep_log_file_num: 1000 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.recycle_log_file_num: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.allow_fallocate: 1 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.allow_mmap_reads: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.allow_mmap_writes: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.use_direct_reads: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.create_missing_column_families: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.db_log_dir: 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.wal_dir: 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-02T11:54:23.252 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.advise_random_on_open: 1 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.db_write_buffer_size: 0 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.write_buffer_manager: 0x5577b7032e60 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.rate_limiter: (nil) 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-02T11:54:23.254 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.wal_recovery_mode: 2 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.enable_thread_tracking: 0 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.enable_pipelined_write: 0 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.unordered_write: 0 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.row_cache: None 2024-09-02T11:54:23.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.wal_filter: None 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.allow_ingest_behind: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.two_write_queues: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.manual_wal_flush: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.wal_compression: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.atomic_flush: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.log_readahead_size: 0 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-02T11:54:23.256 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.best_efforts_recovery: 0 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.allow_data_in_errors: 0 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.db_host_id: __hostname__ 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_background_jobs: 2 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_background_compactions: -1 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_subcompactions: 1 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_total_wal_size: 0 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-02T11:54:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_open_files: -1 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bytes_per_sync: 0 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_readahead_size: 0 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_background_flushes: -1 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Compression algorithms supported: 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kZSTD supported: 0 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kXpressCompression supported: 0 2024-09-02T11:54:23.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kBZip2Compression supported: 0 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kLZ4Compression supported: 1 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kZlibCompression supported: 1 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kLZ4HCCompression supported: 1 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: kSnappyCompression supported: 1 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.merge_operator: 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_filter: None 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_filter_factory: None 2024-09-02T11:54:23.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.sst_partitioner_factory: None 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5577b702f4e0) 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2024-09-02T11:54:23.260 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x5577b7022850 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-09-02T11:54:23.261 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2024-09-02T11:54:23.262 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.write_buffer_size: 33554432 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_write_buffer_number: 2 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression: NoCompression 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression: Disabled 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.prefix_extractor: nullptr 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.num_levels: 7 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-02T11:54:23.263 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.level: 32767 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.strategy: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-02T11:54:23.264 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.enabled: false 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.target_file_size_base: 67108864 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-02T11:54:23.265 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.arena_block_size: 1048576 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.disable_auto_compactions: 0 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-02T11:54:23.266 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.inplace_update_support: 0 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.bloom_locality: 0 2024-09-02T11:54:23.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.max_successive_merges: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.paranoid_file_checks: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.force_consistency_checks: 1 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.report_bg_io_stats: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.ttl: 2592000 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.enable_blob_files: false 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.min_blob_size: 0 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.blob_file_size: 268435456 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-02T11:54:23.268 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.blob_file_starting_level: 0 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b233f977-b65a-449a-95a8-8560843b2ca0 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725278062926109, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725278062926953, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85901, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 261, "table_properties": {"data_size": 84048, "index_size": 237, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10250, "raw_average_key_size": 46, "raw_value_size": 78193, "raw_average_value_size": 355, "num_data_blocks": 11, "num_entries": 220, "num_filter_entries": 220, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725278062, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b233f977-b65a-449a-95a8-8560843b2ca0", "db_session_id": "R3TX8JTIWITAFMIMAQGC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725278062927049, "job": 1, "event": "recovery_finished"} 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-02T11:54:23.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5577b7052e00 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: DB pointer 0x5577b7152000 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-09-02T11:54:23.270 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 85.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 85.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-02T11:54:23.271 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 26.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 26.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x5577b7022850#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-09-02T11:54:23.272 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???) e1 preinit fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).mds e1 new map 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).mds e1 print_map 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: btime 2024-09-02T11:54:18:038386+0000 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-02T11:54:23.273 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: monmap epoch 1 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: last_changed 2024-09-02T11:54:14.838508+0000 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: min_mon_release 19 (squid) 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: election_strategy: 1 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: fsmap 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: osdmap e1: 0 total, 0 up, 0 in 2024-09-02T11:54:23.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 ceph-mon[17163]: mgrmap e1: no daemons active 2024-09-02T11:54:23.275 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 bash[17145]: 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 2024-09-02T11:54:23.275 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:22 smithi064 systemd[1]: Started Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T11:54:24.048 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-02T11:54:24.070 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-02T11:54:24.070 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mgr... 2024-09-02T11:54:24.071 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-02T11:54:24.071 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-02T11:54:24.318 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T11:54:24.318 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x.service: Unit ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x.service not loaded. 2024-09-02T11:54:24.463 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968.target.wants/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service. 2024-09-02T11:54:24.793 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:24 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1275359795' entity='client.admin' 2024-09-02T11:54:24.998 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T11:54:24.998 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-09-02T11:54:25.005 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T11:54:25.005 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-09-02T11:54:25.005 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-09-02T11:54:25.005 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-09-02T11:54:25.014 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T11:54:25.014 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-09-02T11:54:25.021 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T11:54:25.021 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-09-02T11:54:25.021 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-09-02T11:54:25.021 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-02T11:54:25.021 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr to start... 2024-09-02T11:54:25.024 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr... 2024-09-02T11:54:25.094 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:24 smithi064 systemd[1]: Started Ceph mgr.x for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T11:54:25.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:25 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/961869666' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "e8e22764-6921-11ef-bcd4-c7b262605968", 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-02T11:54:26.179 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-02T11:54:26.180 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-02T11:54:26.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-02T11:54:26.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-02T11:54:26.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-02T11:54:26.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-02T11:54:26.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-02T11:54:26.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "btime": "2024-09-02T11:54:18:038386+0000", 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-02T11:54:26.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-09-02T11:54:18.039062+0000", 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:26.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-02T11:54:26.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:54:26.188 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (1/15)... 2024-09-02T11:54:28.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:28 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1449538868' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-02T11:54:29.382 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:29.382 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:54:29.382 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "e8e22764-6921-11ef-bcd4-c7b262605968", 2024-09-02T11:54:29.382 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-02T11:54:29.383 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:29.384 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-02T11:54:29.385 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-02T11:54:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-02T11:54:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-02T11:54:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-02T11:54:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-02T11:54:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-02T11:54:29.386 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "btime": "2024-09-02T11:54:18:038386+0000", 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-02T11:54:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-02T11:54:29.388 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:29.389 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-09-02T11:54:18.039062+0000", 2024-09-02T11:54:29.389 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T11:54:29.389 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:29.389 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-02T11:54:29.389 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:54:29.389 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (2/15)... 2024-09-02T11:54:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: Activating manager daemon x 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: mgrmap e2: x(active, starting, since 0.00328863s) 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: Manager daemon x is now available 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-02T11:54:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' 2024-09-02T11:54:30.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' 2024-09-02T11:54:30.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:30 smithi064 ceph-mon[17163]: from='mgr.14100 172.21.15.64:0/814568880' entity='mgr.x' 2024-09-02T11:54:32.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:32 smithi064 ceph-mon[17163]: mgrmap e3: x(active, since 1.00845s) 2024-09-02T11:54:32.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:32 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/366449635' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "e8e22764-6921-11ef-bcd4-c7b262605968", 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-02T11:54:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:32.609 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-02T11:54:32.610 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-02T11:54:32.610 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-02T11:54:32.614 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "btime": "2024-09-02T11:54:18:038386+0000", 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-02T11:54:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-09-02T11:54:18.039062+0000", 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:54:32.616 INFO:teuthology.orchestra.run.smithi064.stdout:mgr is available 2024-09-02T11:54:33.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:33 smithi064 ceph-mon[17163]: mgrmap e4: x(active, since 2s) 2024-09-02T11:54:33.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:33.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-09-02T11:54:33.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:33.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-02T11:54:33.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-09-02T11:54:33.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-02T11:54:33.714 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling cephadm module... 2024-09-02T11:54:34.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:34 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2483945645' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-02T11:54:35.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:35 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/879262304' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-02T11:54:35.344 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: ignoring --setuser ceph since I am not root 2024-09-02T11:54:35.344 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: ignoring --setgroup ceph since I am not root 2024-09-02T11:54:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:36 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/879262304' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-02T11:54:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:36 smithi064 ceph-mon[17163]: mgrmap e5: x(active, since 5s) 2024-09-02T11:54:36.992 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:54:36.992 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-02T11:54:36.993 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-02T11:54:36.993 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-02T11:54:36.993 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-02T11:54:36.993 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:54:36.993 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-09-02T11:54:36.993 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 5... 2024-09-02T11:54:37.241 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:37 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3110078788' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-02T11:54:40.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: Active manager daemon x restarted 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: Activating manager daemon x 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: osdmap e2: 0 total, 0 up, 0 in 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: mgrmap e6: x(active, starting, since 0.00319039s) 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-02T11:54:40.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:40 smithi064 ceph-mon[17163]: Manager daemon x is now available 2024-09-02T11:54:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:54:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-02T11:54:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-02T11:54:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:54:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 5 is available 2024-09-02T11:54:41.832 INFO:teuthology.orchestra.run.smithi064.stdout:Setting orchestrator backend to cephadm... 2024-09-02T11:54:42.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: mgrmap e7: x(active, since 1.00577s) 2024-09-02T11:54:42.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:42.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: Found migration_current of "None". Setting to last migration. 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-02T11:54:42.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:42 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: 2024-09-02T11:54:43.024+0000 7f1601b89640 -1 log_channel(cephadm) log [ERR] : [02/Sep/2024:11:54:43] ENGINE Error in HTTPServer.serve 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: Traceback (most recent call last): 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self._connections.run(self.expiration_interval) 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self._run(expiration_interval) 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T11:54:43.078 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: s = self.context.wrap_socket( 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: return self.sslsocket_class._create( 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self.do_handshake() 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self._sslobj.do_handshake() 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T11:54:43.079 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: mgrmap e8: x(active, since 2s) 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: [02/Sep/2024:11:54:42] ENGINE Bus STARTING 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: [02/Sep/2024:11:54:43] ENGINE Serving on https://172.21.15.64:7150 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: [02/Sep/2024:11:54:43] ENGINE Error in HTTPServer.serve 2024-09-02T11:54:43.671 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._connections.run(self.expiration_interval) 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._run(expiration_interval) 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: s = self.context.wrap_socket( 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T11:54:43.672 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.sslsocket_class._create( 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout: self.do_handshake() 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._sslobj.do_handshake() 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: [02/Sep/2024:11:54:43] ENGINE Serving on http://172.21.15.64:8765 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: [02/Sep/2024:11:54:43] ENGINE Bus STARTED 2024-09-02T11:54:43.673 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:43 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:54:44.336 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-02T11:54:44.337 INFO:teuthology.orchestra.run.smithi064.stdout:Generating ssh key... 2024-09-02T11:54:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:44 smithi064 ceph-mon[17163]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:45.344 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: Generating public/private rsa key pair. 2024-09-02T11:54:45.344 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: Your identification has been saved in /tmp/tmp9umkis3l/key 2024-09-02T11:54:45.344 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: Your public key has been saved in /tmp/tmp9umkis3l/key.pub 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: The key fingerprint is: 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: SHA256:AT6o8haqgMY5MAAHJSTnT7LsA823Nx5Scy+rfk86Afs ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: The key's randomart image is: 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: +---[RSA 3072]----+ 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |*++ . | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |o= o . | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |. o o o . | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |.+ * .. . | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |= B o ooS | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |+B.o o.o.. | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |+== o +...o | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |+... + oE= | 2024-09-02T11:54:45.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: |. .+ooo. | 2024-09-02T11:54:45.346 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: +----[SHA256]-----+ 2024-09-02T11:54:46.134 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:46 smithi064 ceph-mon[17163]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:46.134 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:46 smithi064 ceph-mon[17163]: Generating ssh key... 2024-09-02T11:54:46.134 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:46 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:46.135 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:46 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:46.924 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:54:46.924 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-02T11:54:46.924 INFO:teuthology.orchestra.run.smithi064.stdout:Adding key to root@localhost authorized_keys... 2024-09-02T11:54:46.925 INFO:teuthology.orchestra.run.smithi064.stdout:Adding host smithi064... 2024-09-02T11:54:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:47 smithi064 ceph-mon[17163]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:48.224 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:48 smithi064 ceph-mon[17163]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:49.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:49 smithi064 ceph-mon[17163]: Deploying cephadm binary to smithi064 2024-09-02T11:54:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Added host 'smithi064' with addr '172.21.15.64' 2024-09-02T11:54:52.040 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mon service... 2024-09-02T11:54:52.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:52 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:52.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:52 smithi064 ceph-mon[17163]: Added host smithi064 2024-09-02T11:54:52.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:52 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:54:53.551 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-02T11:54:53.552 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mgr service... 2024-09-02T11:54:53.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:53 smithi064 ceph-mon[17163]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:53.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:53 smithi064 ceph-mon[17163]: Saving service mon spec with placement count:5 2024-09-02T11:54:53.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:53 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:55.478 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:55 smithi064 ceph-mon[17163]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:54:55.478 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:55 smithi064 ceph-mon[17163]: Saving service mgr spec with placement count:2 2024-09-02T11:54:55.478 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:55 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:55.478 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:55 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:55.478 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:55 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:55.480 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-02T11:54:57.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:57 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2998289044' entity='client.admin' 2024-09-02T11:54:58.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:58 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1429690132' entity='client.admin' 2024-09-02T11:54:58.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:58 smithi064 ceph-mon[17163]: from='mgr.14118 172.21.15.64:0/4255680598' entity='mgr.x' 2024-09-02T11:54:58.337 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling the dashboard module... 2024-09-02T11:54:59.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:54:59 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3494131367' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-02T11:54:59.594 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: ignoring --setuser ceph since I am not root 2024-09-02T11:54:59.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:54:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: ignoring --setgroup ceph since I am not root 2024-09-02T11:55:00.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:00 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3494131367' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-02T11:55:00.529 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:00 smithi064 ceph-mon[17163]: mgrmap e9: x(active, since 18s) 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-09-02T11:55:01.110 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 9... 2024-09-02T11:55:01.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:01 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2327455452' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-02T11:55:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: Active manager daemon x restarted 2024-09-02T11:55:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: Activating manager daemon x 2024-09-02T11:55:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: osdmap e3: 0 total, 0 up, 0 in 2024-09-02T11:55:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: mgrmap e10: x(active, starting, since 0.00366527s) 2024-09-02T11:55:04.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:55:04.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-02T11:55:04.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mds metadata"} : dispatch 2024-09-02T11:55:04.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata"} : dispatch 2024-09-02T11:55:04.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata"} : dispatch 2024-09-02T11:55:04.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:04 smithi064 ceph-mon[17163]: Manager daemon x is now available 2024-09-02T11:55:05.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:05 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:05.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:05 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"} : dispatch 2024-09-02T11:55:05.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:05 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"} : dispatch 2024-09-02T11:55:05.763 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:05 smithi064 ceph-mon[17163]: mgrmap e11: x(active, since 1.00618s) 2024-09-02T11:55:06.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-09-02T11:55:06.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-02T11:55:06.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-02T11:55:06.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-09-02T11:55:06.049 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 9 is available 2024-09-02T11:55:06.049 INFO:teuthology.orchestra.run.smithi064.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-02T11:55:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-mon[17163]: [02/Sep/2024:11:55:06] ENGINE Bus STARTING 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: 2024-09-02T11:55:06.320+0000 7fac46f17640 -1 log_channel(cephadm) log [ERR] : [02/Sep/2024:11:55:06] ENGINE Error in HTTPServer.serve 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: Traceback (most recent call last): 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self._connections.run(self.expiration_interval) 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self._run(expiration_interval) 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T11:55:06.595 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: s = self.context.wrap_socket( 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: return self.sslsocket_class._create( 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self.do_handshake() 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: self._sslobj.do_handshake() 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T11:55:06.596 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 11:55:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: 2024-09-02T11:55:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: [02/Sep/2024:11:55:06] ENGINE Serving on https://172.21.15.64:7150 2024-09-02T11:55:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: [02/Sep/2024:11:55:06] ENGINE Error in HTTPServer.serve 2024-09-02T11:55:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout: Traceback (most recent call last): 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._connections.run(self.expiration_interval) 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._run(expiration_interval) 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T11:55:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: s = self.context.wrap_socket( 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: return self.sslsocket_class._create( 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: self.do_handshake() 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: self._sslobj.do_handshake() 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T11:55:07.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: [02/Sep/2024:11:55:06] ENGINE Serving on http://172.21.15.64:8765 2024-09-02T11:55:07.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: [02/Sep/2024:11:55:06] ENGINE Bus STARTED 2024-09-02T11:55:07.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: mgrmap e12: x(active, since 2s) 2024-09-02T11:55:07.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:07.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:07 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:09.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:09.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUIJ5zxmMZX3lEJQZnWZTtnhq8800wDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkwMjExNTUwOFoXDTM0\nMDkwMzExNTUwOFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjY0MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAvp9Y4YXjAqmBVcZWYFYI4nK76NRna78CbO0y\nyC+7ZmAQEVM1wHhcJWs7EO4hRaoJpW/CRCrCDE+jeQ0YCs4dbcgEUVpk3sSczlaH\nkKti6V4WUr/ffEF6269wY3U1vYDLU5d9JckGy3AZ3K0QBRt+5aA+SDqnwYaLGdh2\n2Ro5+lJ0Z2zGdzr8zFu1uGmRjoZ3EgX8QqPpteO1fHzogfDKmuIQHERv/xgZTwhB\nJXK2VPGm7fm5TwpS/MGmHHWT5HyAbW1x+bMHy1mGFlBtxfy+5zvGMBSshT8sn3LK\nTitfjcU8ecEplB16limXrOY1CVy8XlR9RxhJc0gGe577ts0iOjkRmLEVTCnDvPbC\nja5tW1hcIzZmh+U28Cr2ywLcI0PxjEX+E6JiUk0vQdo79hljvLB92bbyiRAkjt/P\nfVtQdZcQj+KCMjPaSxI0lSpzra36V4nSanz3iXWkEfxu9Dy3KOnzdiBvAGFDsyaO\nmkWeXiyRqY1Cz37BsYfN401dAnia7v7mDvYavon4OWlfjOaVmuOrrFzzZijDg0mo\nOQUWOJt7geP8ImkMYjgjqpbovdnS8uOBL8RT+Aq2eF5MUsw4+em6PorhzXl7gz90\nB3omeieVq2zgV2VlPE3mKvQWcPQ0A40/iqlLtQs2sGipiXwCpiYhl0D2rm6GasS/\noYvIGX0CAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDY0ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPQDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQBh\nLzxOqrgX0v3jsf22rbkRaKq9ZCqaTChG1Ciw7xLeBX6QEWMAICqkdtOXSyFgXRme\nd1ASShHXJMuCO4D5vNbzk05e0c2wVIf0eV2Ke20ubblFL9cY7bQYGEDxTrDL5dvL\ntIKJDLYfJe977UFg0qIS7jCBsy8HW+opi3coggNdzb+0sEck3+7EtEszF1Ln9NhK\n9WXtxYQ6f90+dsQKj6o0xgkpq7dBfYawfuYSJvRfdWMFG6Vq7RMCxs1chEssEnJ2\nfNDKhUSQaBMdBI0frklII0VbzxFo5EmA8FarYOyvMkgzHv6y87TDYlBEGpgrGXMf\n8Qo1H+cWjlnnmCYhZqTNoe3fO/ck2mHWH0hxYHIgunhKjh6TN/XeMFA6ZIKdNsxq\n7dErzybx9fwmlwsuF8GKpUS15DNv/8RXMbmDMv1fw10EYycMPfkV06Eed7veGZXC\niePTVGXfYMdk8KPOBGMCcK0YZxJF3fiVi1wRQ+v3EDss57hSqm1lKflg+X/f2NbG\nk7EZwg1vB/ZRlqZCT8xyAcNnkC4601EooNfvXw96B32sdpyzbNvy0NggcChh8smL\nloxZlQmU7xNYze3jgI2yeWFh9JyG56WzSIv6XdMoChTDi3AE7PGlIFgV80Lv8dGL\n9xJ+v4qNTsKpqDVfS5LGgghC2RTj7gWpxa8TeLw0BA==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAvp9Y4YXjAqmBVcZWYFYI4nK76NRna78CbO0yyC+7ZmAQEVM1\nwHhcJWs7EO4hRaoJpW/CRCrCDE+jeQ0YCs4dbcgEUVpk3sSczlaHkKti6V4WUr/f\nfEF6269wY3U1vYDLU5d9JckGy3AZ3K0QBRt+5aA+SDqnwYaLGdh22Ro5+lJ0Z2zG\ndzr8zFu1uGmRjoZ3EgX8QqPpteO1fHzogfDKmuIQHERv/xgZTwhBJXK2VPGm7fm5\nTwpS/MGmHHWT5HyAbW1x+bMHy1mGFlBtxfy+5zvGMBSshT8sn3LKTitfjcU8ecEp\nlB16limXrOY1CVy8XlR9RxhJc0gGe577ts0iOjkRmLEVTCnDvPbCja5tW1hcIzZm\nh+U28Cr2ywLcI0PxjEX+E6JiUk0vQdo79hljvLB92bbyiRAkjt/PfVtQdZcQj+KC\nMjPaSxI0lSpzra36V4nSanz3iXWkEfxu9Dy3KOnzdiBvAGFDsyaOmkWeXiyRqY1C\nz37BsYfN401dAnia7v7mDvYavon4OWlfjOaVmuOrrFzzZijDg0moOQUWOJt7geP8\nImkMYjgjqpbovdnS8uOBL8RT+Aq2eF5MUsw4+em6PorhzXl7gz90B3omeieVq2zg\nV2VlPE3mKvQWcPQ0A40/iqlLtQs2sGipiXwCpiYhl0D2rm6GasS/oYvIGX0CAwEA\nAQKCAgAM0Szk1b+0FIl7puSfa8wt97jOo1uEg+PJ8bAArwRFfQTk4/4A//Fy2vvY\n0VN/j5BAUAcAAN+qfBSvOr6+A+pi0OMIxPsefG9ZCLnq87nTRwOEEwktSGornnyq\n1KfP+GU+LLkvjXhYEk9MMSnFgiSylBH5VtJOoaPgPBe6VM2xrmd2cyVWwSqxeKLR\nGxi8UdUXQJDxOnwS12mtohNwdexAOkjgMbIrH5JyEoluLGv57a6uS88r3fNDB1Kf\n5sBSib7ekhBAEcuJdC6fI6TMk3LoRt1hAGmHjb4hHCg3VdTCClFHlCyeHCumMEvr\nG6Z/cj1rL/ildgqtm5nt/XxffU8FUoGXl9+lRK3mHLxprPntLF2Tadz0vseTkT+3\nIvEv452eURxTHLTsrKv5qO6EMBIm6GhRQrpG7/uXniO+e+G0027qTFxxVKFURxfB\nVKR2Dj+6L7Lqdtlq82P2X4X6Djrh/etgDLSIQOWqpBInQsBvCfY7d0f64P1meJnG\nSIEq3EmlorVmR0TOnTL0y8JuCC8DwHZnxx1tBv4Q2MXE28vkUUvfTq7quIdmRn2r\nIgzglr7K3xPTTdzGx7iWomFJOmyI/TYCq0SogZtGZiTQYfEVhOmAyaAbeV9XJ6+y\nEUXjVHY3E/Sos9p/3pMEickMNhAsv2cFiK3Vr9ULrpicppjtgQKCAQEA8x2aDiNF\n0S5wRRFdeIUEoHd7kScFdwXYw5XAD0beyxtTSjhnuqcN/UmXO7vh9CR/Zn9cKB7E\no2ddkJouUfRaRjNqzTvRbZSSBhvwKlpOmCPC7jVnGfac6eZyKKWa5+x6U4gLoNZQ\n7hWQx7yEsFHFv2/l5zt9SSDDtccc8IcBan2fX9hgKuaulG9ALW/trBW6nQPee6Jx\nQXfOoKKfM7aL26bAW8oK1kUnX2GuuEzafBiFdjSG+y3ytbm8Tkp6Qc4bplhmAFf9\nGhn1XduavAUhcNcQBN6Edc/tn3eW+Koosqhl4yQ0KNUjec9dNYvLR44CInScrttY\n9Sa7wNdtuDDq/QKCAQEAyLmPWRkG+xCePKsPQgJ0EX6zk+8G+YrwcSWUYpua0noP\nF+usB3zoq+asbiowKVoehP2sFZ/iQEZ9PZCqW/llcRK5VH9JDZMWHW5Lnh6gj4Gh\nUgZ+kP9WiCJa8S0+ThmRGfRD4oW+dQAVCJOBq1+nLId1bsOd47fNIKENdtnOt/Ch\nfYBpSbgq0WaZ3lY9R11lSlmTPemaN578AVzrZ0wZkctbopssBWR5nqlJTVyF0kYD\nVXgXlgP2zvNZ3Bqi6+Ko/E7XXjAl3Pr8w9ByMs0gMA9XrzLnTdzXBhEs/whEEn7d\n2DXvqJmOzxyePF9bRcUKXQeEv1nY0SWAM7ACpY9wgQKCAQAEqqrMXqz2aPfKu5yZ\ntaPzGJgMnz0hncEdcABxyqB4MZwVrqN/F7nTBNicUuzxbMntQ8CXHJAVuuQ25F7D\nP/eDpXUFX4KToehzqZGgnn8nuHgDbo7+FpfIfZ3vXc5taUKn29KtUa/YAh33V40M\nfWVYWWOGXsP25WQcVygdDdpfGu42PscJy8fKHaCTvrRHO1uLrAlXQjT+VrhmqPh5\nW6Wvn7HNkUFH/6fT0uIxs3f2LdpR/Iwiwi0Ig/ozXoaeQMCTHttGYmn/WEQ3ewGr\nXkXYDqjGDydnw4gCKZ+CQnHpjAX4qDdAGWdG7U4Suw4532X3F1nymU9J1WaBOyFm\nCbpZAoIBAQC+fTMoGBNC4sN4ydICD/f8vmRH6RiSa6PBjStj5U4pnwKB7ftlY8of\nkRtktQ7dv5OlHGRaZVA/NWYXrhmujWESzPUq1l0Kz+K17X0/wssXRatF/7nPezKw\noQHzTb+KOMM2gVSHPevAd2J+sBXoAC/Tl33/rw9BE6ZM5+qyNwPyFh/7zYieGHQy\nI4+0q90dw+mjhJyyreXYET4RQRB6CINflEETcPV27KJqyzcwfrTVF/KEAyNHD94D\nCnjl5DVnbMR4JY5H6aOVMm/8mO43femlEnphXwqaHxHExToRkdOgRdvdG1qkg44+\nyWb13nztGpC6Nq9pNSgJW/BtnQT1/8mBAoIBAQDaPLA0mxn/wHw0B3915fPJWHIa\nSTCeEzX5QWks1BIb0IHoPK6nRG8il1EM0a320z8DYVT43oBAU2HUKP8yaJpm97ox\n7ek2xgLnyrHITcahEPtMhta6VLU7g2MciMjDuN41MtdO40WYsihpsVCvR7xkKpwI\nrWDkz8WGOwKp7g9bo8wfuttVGan4R+ahc+mbks3e6NUaynjN37v7eNQQChUe2QKX\nd77Em2+73J/1/bM+xfGM/Chvx/ahHxmeyJrz/YrVY0sfrTiK6h81PvcqTBrpohaT\nNbtmG1velCSs8EOtQXhZhUtuUwEzUkhHaREhY6PEOOXiAlsbFTmspYw8BpdM\n-----END RSA PRIVATE KEY-----\n"} 2024-09-02T11:55:10.971 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-02T11:55:11.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:11 smithi064 ceph-mon[17163]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:11.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:12.177 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-02T11:55:12.178 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial admin user... 2024-09-02T11:55:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:12 smithi064 ceph-mon[17163]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:13.034 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:13.667 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$2PtywW65sUcQqT0mFNUb2OaxbcewojDSeHwmnC2dgdF19awq3Xbiy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725278113, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-02T11:55:13.667 INFO:teuthology.orchestra.run.smithi064.stdout:Fetching dashboard port number... 2024-09-02T11:55:14.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:14 smithi064 ceph-mon[17163]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:14.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:14.822 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 8443 2024-09-02T11:55:14.833 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T11:55:14.833 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-09-02T11:55:14.840 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T11:55:14.840 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-09-02T11:55:14.840 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-09-02T11:55:14.840 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph Dashboard is now available at: 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout: URL: https://smithi064:8443/ 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout: User: admin 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout: Password: 0q6s8vegt7 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:14.842 INFO:teuthology.orchestra.run.smithi064.stdout:Saving cluster configuration to /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config directory 2024-09-02T11:55:15.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:15 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/106642980' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-02T11:55:16.122 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-02T11:55:16.122 INFO:teuthology.orchestra.run.smithi064.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout:Or, if you are only running a single cluster on this host: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: ceph telemetry on 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.123 INFO:teuthology.orchestra.run.smithi064.stdout:For more information see: 2024-09-02T11:55:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-02T11:55:16.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:16.124 INFO:teuthology.orchestra.run.smithi064.stdout:Bootstrap complete. 2024-09-02T11:55:16.163 INFO:tasks.cephadm:Fetching config... 2024-09-02T11:55:16.163 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:55:16.163 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-02T11:55:16.179 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-02T11:55:16.179 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:55:16.179 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-02T11:55:16.234 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-02T11:55:16.234 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:55:16.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-02T11:55:16.298 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-02T11:55:16.299 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:55:16.299 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-02T11:55:16.355 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-02T11:55:16.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T11:55:16.395 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:16.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T11:55:16.455 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:16.466 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T11:55:16.508 INFO:teuthology.orchestra.run.smithi087.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:16.520 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T11:55:16.562 INFO:teuthology.orchestra.run.smithi094.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:16.574 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T11:55:16.616 INFO:teuthology.orchestra.run.smithi099.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCT+ht81YXVO1pPImuw30T9af7o9TCadZ29qc+N6DPtV2rEWOOs4UxWeSikOs4wpPn3SEp8ZfbNY8yK8f54QCVCZ287eAGWoseCmdpARq3AdOtPYijX/Z8barPUlK3AIeOivHG2B3UOHGZyqPenUNkWicF+xGo7apSXMnlmBBNr9mcS2cl0xTFPJpcMMUgFJzD4184ZPiR8K6k7QARPQZgTugVme8W0dDJ+wTijJKrjHMIh7gvraXbgWigGZQ6B1TfXGyYb6doAyqC/LIHM/p9h9adUry2b2C2bBu+Yoyvsrnu3B6TiBQhQ8F6D+Xyv/M0+U1u2DldmPpoUqjyfWmuWe7FRi7q4L7orHBXfWU38eQmXc59tQ0ecXd1/1wnvKQjI2ZYyKrSrnm+daWrPphMd8JnAG7w9gvh6K9MpK90cpq8qoC2VBvOg7bcJV3ymykIYFtccoDuR3roYJTHdjz7IH9ej4C6WkuDgqFb+YkVE7rOyz7YHYQJTt9TnWwwxmK0= ceph-e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:16.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-02T11:55:16.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:16 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1223030692' entity='client.admin' 2024-09-02T11:55:16.881 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:17 smithi064 ceph-mon[17163]: mgrmap e13: x(active, since 12s) 2024-09-02T11:55:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:17 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1643968575' entity='client.admin' 2024-09-02T11:55:18.645 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-02T11:55:18.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-02T11:55:18.893 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:19.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:19.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:19.269 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:19.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:19.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:19.270 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:20.276 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi038 2024-09-02T11:55:20.276 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:55:20.276 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.conf 2024-09-02T11:55:20.296 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:55:20.296 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:20.353 INFO:tasks.cephadm:Adding host smithi038 to orchestrator... 2024-09-02T11:55:20.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host add smithi038 2024-09-02T11:55:20.598 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:20.718 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:20.718 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:20.718 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:20.718 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: Updating smithi064:/etc/ceph/ceph.conf 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: Updating smithi064:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: Updating smithi064:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:20.719 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:21.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:21 smithi064 ceph-mon[17163]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:23.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:23 smithi064 ceph-mon[17163]: Deploying cephadm binary to smithi038 2024-09-02T11:55:25.107 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi038' with addr '172.21.15.38' 2024-09-02T11:55:25.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-09-02T11:55:25.976 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:26.426 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:26 smithi064 ceph-mon[17163]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:26.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:26.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:26 smithi064 ceph-mon[17163]: Added host smithi038 2024-09-02T11:55:26.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:26.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:26.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:26.613 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:26.613 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}] 2024-09-02T11:55:27.233 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi087 2024-09-02T11:55:27.233 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:55:27.234 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/etc/ceph/ceph.conf 2024-09-02T11:55:27.253 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:55:27.253 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:27.312 INFO:tasks.cephadm:Adding host smithi087 to orchestrator... 2024-09-02T11:55:27.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host add smithi087 2024-09-02T11:55:27.556 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:27.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:27 smithi064 ceph-mon[17163]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:27.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:27 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:27.770 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:27 smithi064 ceph-mon[17163]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:28.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:28 smithi064 ceph-mon[17163]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: Updating smithi038:/etc/ceph/ceph.conf 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:30.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:31.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:31 smithi064 ceph-mon[17163]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:31.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:31 smithi064 ceph-mon[17163]: Deploying cephadm binary to smithi087 2024-09-02T11:55:31.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:31 smithi064 ceph-mon[17163]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:31.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:31 smithi064 ceph-mon[17163]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:32.195 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi087' with addr '172.21.15.87' 2024-09-02T11:55:32.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:32 smithi064 ceph-mon[17163]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:32.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-09-02T11:55:33.056 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:33.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:33.497 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:33 smithi064 ceph-mon[17163]: Added host smithi087 2024-09-02T11:55:33.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:33.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:33.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:33.693 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:33.693 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}] 2024-09-02T11:55:34.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:34 smithi064 ceph-mon[17163]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:34.261 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:34.262 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi094 2024-09-02T11:55:34.262 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:55:34.262 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/etc/ceph/ceph.conf 2024-09-02T11:55:34.281 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:55:34.281 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:34.340 INFO:tasks.cephadm:Adding host smithi094 to orchestrator... 2024-09-02T11:55:34.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host add smithi094 2024-09-02T11:55:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:35.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:35 smithi064 ceph-mon[17163]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:36 smithi064 ceph-mon[17163]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:36 smithi064 ceph-mon[17163]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:37.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: Deploying cephadm binary to smithi094 2024-09-02T11:55:37.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: Updating smithi087:/etc/ceph/ceph.conf 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:37.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:37.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:37.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:39.126 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi094' with addr '172.21.15.94' 2024-09-02T11:55:39.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-09-02T11:55:39.993 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:40.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:40 smithi064 ceph-mon[17163]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:40.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:40 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:40.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:40 smithi064 ceph-mon[17163]: Added host smithi094 2024-09-02T11:55:40.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:40 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:40.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:40 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:40.351 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:40 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:40.630 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:40.631 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}, {"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}] 2024-09-02T11:55:41.233 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi099 2024-09-02T11:55:41.234 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:55:41.234 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/etc/ceph/ceph.conf 2024-09-02T11:55:41.253 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:55:41.254 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:41.311 INFO:tasks.cephadm:Adding host smithi099 to orchestrator... 2024-09-02T11:55:41.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host add smithi099 2024-09-02T11:55:41.561 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:41.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:41 smithi064 ceph-mon[17163]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:41.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:41 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:41.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:41 smithi064 ceph-mon[17163]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:42.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:42 smithi064 ceph-mon[17163]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: Updating smithi094:/etc/ceph/ceph.conf 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:45 smithi064 ceph-mon[17163]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:45 smithi064 ceph-mon[17163]: Deploying cephadm binary to smithi099 2024-09-02T11:55:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:45 smithi064 ceph-mon[17163]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:45 smithi064 ceph-mon[17163]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:46.187 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi099' with addr '172.21.15.99' 2024-09-02T11:55:46.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:46 smithi064 ceph-mon[17163]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:46.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-09-02T11:55:47.063 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:47.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:47.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:47 smithi064 ceph-mon[17163]: Added host smithi099 2024-09-02T11:55:47.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:47.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:47.464 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:47.688 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:55:47.688 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}, {"addr": "172.21.15.94", "hostname": "smithi094", "labels": [], "status": ""}, {"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}] 2024-09-02T11:55:48.309 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:48 smithi064 ceph-mon[17163]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:48.309 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:48.338 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-02T11:55:48.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd crush tunables default 2024-09-02T11:55:48.582 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:55:49.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:49 smithi064 ceph-mon[17163]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:50.198 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-09-02T11:55:50.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:50 smithi064 ceph-mon[17163]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:50.532 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:50 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/4081470598' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-02T11:55:50.869 INFO:tasks.cephadm:Adding mon.a on smithi064 2024-09-02T11:55:50.869 INFO:tasks.cephadm:Adding mon.b on smithi087 2024-09-02T11:55:50.870 INFO:tasks.cephadm:Adding mon.c on smithi094 2024-09-02T11:55:50.870 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch apply mon '3;smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c' 2024-09-02T11:55:51.112 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:51.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/4081470598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-02T11:55:51.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:55:51.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:51.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:51.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:51.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:51.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:51.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:51.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:51.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:51.737 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled mon update... 2024-09-02T11:55:52.328 DEBUG:teuthology.orchestra.run.smithi087:mon.b> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b.service 2024-09-02T11:55:52.331 DEBUG:teuthology.orchestra.run.smithi094:mon.c> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c.service 2024-09-02T11:55:52.334 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T11:55:52.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T11:55:52.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:55:52.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:52.670 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:53.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:53 smithi064 ceph-mon[17163]: from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:53.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:53 smithi064 ceph-mon[17163]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c;count:3 2024-09-02T11:55:53.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:53 smithi064 ceph-mon[17163]: Deploying daemon mon.c on smithi094 2024-09-02T11:55:53.836 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:55:53.837 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":1,"fsid":"e8e22764-6921-11ef-bcd4-c7b262605968","modified":"2024-09-02T11:54:14.838508Z","created":"2024-09-02T11:54:14.838508Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-02T11:55:53.837 INFO:teuthology.orchestra.run.smithi094.stderr:dumped monmap epoch 1 2024-09-02T11:55:54.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:54 smithi064 ceph-mon[17163]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:54.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:55:54 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/1043025568' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:55:55.766 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T11:55:55.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T11:55:56.048 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 podman[20354]: 2024-09-02 11:55:56.025770918 +0000 UTC m=+0.337601053 container start 9b7c8f05dfb3be4c5d8aab86080b4e765b816a2f679a1112952d8b975717a5e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-c, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux ) 2024-09-02T11:55:56.199 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: set uid:gid to 167:167 (ceph:ceph) 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: ceph version 19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev), process ceph-mon, pid 2 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pidfile_write: ignore empty --pid-file 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: load: jerasure load: lrc 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: RocksDB version: 7.9.2 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Git sha 0 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Compile date 2024-08-29 08:01:36 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: DB SUMMARY 2024-09-02T11:55:56.317 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: DB Session ID: 8IFKBTJK6Q19SXHG6D53 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: CURRENT file: CURRENT 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: IDENTITY file: IDENTITY 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.error_if_exists: 0 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.create_if_missing: 0 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.paranoid_checks: 1 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-02T11:55:56.318 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.env: 0x55a074977160 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.fs: PosixFileSystem 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.info_log: 0x55a076429880 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_file_opening_threads: 16 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.statistics: (nil) 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.use_fsync: 0 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_log_file_size: 0 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.keep_log_file_num: 1000 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.recycle_log_file_num: 0 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.allow_fallocate: 1 2024-09-02T11:55:56.319 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.allow_mmap_reads: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.allow_mmap_writes: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.use_direct_reads: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.create_missing_column_families: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.db_log_dir: 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.wal_dir: 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-02T11:55:56.320 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.advise_random_on_open: 1 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.db_write_buffer_size: 0 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.write_buffer_manager: 0x55a07642ce60 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.rate_limiter: (nil) 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.wal_recovery_mode: 2 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.enable_thread_tracking: 0 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.enable_pipelined_write: 0 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.unordered_write: 0 2024-09-02T11:55:56.321 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.row_cache: None 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.wal_filter: None 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.allow_ingest_behind: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.two_write_queues: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.manual_wal_flush: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.wal_compression: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.atomic_flush: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-02T11:55:56.322 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.log_readahead_size: 0 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.best_efforts_recovery: 0 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.allow_data_in_errors: 0 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.db_host_id: __hostname__ 2024-09-02T11:55:56.323 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_background_jobs: 2 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_background_compactions: -1 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_subcompactions: 1 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_total_wal_size: 0 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-02T11:55:56.325 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_open_files: -1 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bytes_per_sync: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_readahead_size: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_background_flushes: -1 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Compression algorithms supported: 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kZSTD supported: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kXpressCompression supported: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kBZip2Compression supported: 0 2024-09-02T11:55:56.326 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kLZ4Compression supported: 1 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kZlibCompression supported: 1 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kLZ4HCCompression supported: 1 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: kSnappyCompression supported: 1 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.merge_operator: 2024-09-02T11:55:56.327 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_filter: None 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_filter_factory: None 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.sst_partitioner_factory: None 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a0764294e0) 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: cache_index_and_filter_blocks: 1 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: pin_top_level_index_and_filter: 1 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: index_type: 0 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: data_block_index_type: 0 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: index_shortening: 1 2024-09-02T11:55:56.328 INFO:journalctl@ceph.mon.c.smithi094.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: checksum: 4 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: no_block_cache: 0 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache: 0x55a07641c850 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache_name: BinnedLRUCache 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache_options: 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: capacity : 536870912 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: num_shard_bits : 4 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: strict_capacity_limit : 0 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: high_pri_pool_ratio: 0.000 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_cache_compressed: (nil) 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: persistent_cache: (nil) 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_size: 4096 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_size_deviation: 10 2024-09-02T11:55:56.329 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_restart_interval: 16 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: index_block_restart_interval: 1 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: metadata_block_size: 4096 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: partition_filters: 0 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: use_delta_encoding: 1 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: filter_policy: bloomfilter 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: whole_key_filtering: 1 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: verify_compression: 0 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: read_amp_bytes_per_bit: 0 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: format_version: 5 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: enable_index_compression: 1 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: block_align: 0 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: max_auto_readahead_size: 262144 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: prepopulate_block_cache: 0 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: initial_auto_readahead_size: 8192 2024-09-02T11:55:56.330 INFO:journalctl@ceph.mon.c.smithi094.stdout: num_file_reads_for_auto_readahead: 2 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.write_buffer_size: 33554432 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_write_buffer_number: 2 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression: NoCompression 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression: Disabled 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.prefix_extractor: nullptr 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.num_levels: 7 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-02T11:55:56.331 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.level: 32767 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.strategy: 0 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.enabled: false 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-02T11:55:56.332 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.target_file_size_base: 67108864 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-02T11:55:56.333 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.arena_block_size: 1048576 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.disable_auto_compactions: 0 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-02T11:55:56.334 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.inplace_update_support: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.bloom_locality: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.max_successive_merges: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-02T11:55:56.335 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.paranoid_file_checks: 0 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.force_consistency_checks: 1 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.report_bg_io_stats: 0 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.ttl: 2592000 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.enable_blob_files: false 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.min_blob_size: 0 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.blob_file_size: 268435456 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-02T11:55:56.336 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.blob_file_starting_level: 0 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7dc93ffb-376c-4e7a-8e1d-0644b0bccf06 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725278156052917, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725278156053312, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725278156, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7dc93ffb-376c-4e7a-8e1d-0644b0bccf06", "db_session_id": "8IFKBTJK6Q19SXHG6D53", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725278156053404, "job": 1, "event": "recovery_finished"} 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-02T11:55:56.337 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a07644ce00 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: DB pointer 0x55a07654c000 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** DB Stats ** 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T11:55:56.338 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** Compaction Stats [default] ** 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** Compaction Stats [default] ** 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-02T11:55:56.339 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-02T11:55:56.340 INFO:journalctl@ceph.mon.c.smithi094.stdout: Block cache BinnedLRUCache@0x55a07641c850#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: using public_addr v2:172.21.15.94:0/0 -> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: starting mon.c rank -1 at public addrs [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] at bind addrs [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(???) e0 preinit fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).mds e1 new map 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-02T11:55:56.341 INFO:journalctl@ceph.mon.c.smithi094.stdout: e1 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout: btime 2024-09-02T11:54:18:038386+0000 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout: legacy client fscid: -1 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout: 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout: No filesystems configured 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-02T11:55:56.342 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Deploying cephadm binary to smithi087 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Added host smithi087 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.343 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Deploying cephadm binary to smithi094 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:56.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi087:/etc/ceph/ceph.conf 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Added host smithi094 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi094:/etc/ceph/ceph.conf 2024-09-02T11:55:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Deploying cephadm binary to smithi099 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Added host smithi099 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/4081470598' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/4081470598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"} : dispatch 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:55:56.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c;count:3 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: Deploying daemon mon.c on smithi094 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/1043025568' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 ceph-mon[20404]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 bash[20354]: 9b7c8f05dfb3be4c5d8aab86080b4e765b816a2f679a1112952d8b975717a5e5 2024-09-02T11:55:56.350 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:55:56 smithi094 systemd[1]: Started Ceph mon.c for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: mon.a calling monitor election 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: mon.c calling monitor election 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: monmap epoch 2 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: last_changed 2024-09-02T11:55:56.078299+0000 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: min_mon_release 19 (squid) 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: election_strategy: 1 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: 1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.c 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: fsmap 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: mgrmap e13: x(active, since 56s) 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: overall HEALTH_OK 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:01.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: mon.a calling monitor election 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: mon.c calling monitor election 2024-09-02T11:56:01.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: monmap epoch 2 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: last_changed 2024-09-02T11:55:56.078299+0000 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: min_mon_release 19 (squid) 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: election_strategy: 1 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: 1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.c 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: fsmap 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:01.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: mgrmap e13: x(active, since 56s) 2024-09-02T11:56:01.599 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: overall HEALTH_OK 2024-09-02T11:56:01.599 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:01.599 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:01.599 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:01.599 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:01.599 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:02.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:02 smithi064 ceph-mon[17163]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:02.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:02 smithi064 ceph-mon[17163]: Deploying daemon mon.b on smithi087 2024-09-02T11:56:02.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:02.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:02 smithi094 ceph-mon[20404]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:02.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:02 smithi094 ceph-mon[20404]: Deploying daemon mon.b on smithi087 2024-09-02T11:56:02.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:02.900 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:56:02.900 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":2,"fsid":"e8e22764-6921-11ef-bcd4-c7b262605968","modified":"2024-09-02T11:55:56.078299Z","created":"2024-09-02T11:54:14.838508Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-02T11:56:02.900 INFO:teuthology.orchestra.run.smithi094.stderr:dumped monmap epoch 2 2024-09-02T11:56:03.413 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:03 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/2678507150' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:03.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:03 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/2678507150' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:04.441 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T11:56:04.441 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T11:56:04.470 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:04 smithi094 ceph-mon[20404]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:04.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:04 smithi064 ceph-mon[17163]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:04.694 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:56:04.921 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).mds e1 new map 2024-09-02T11:56:04.921 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-02T11:56:04.921 INFO:journalctl@ceph.mon.b.smithi087.stdout: e1 2024-09-02T11:56:04.921 INFO:journalctl@ceph.mon.b.smithi087.stdout: btime 2024-09-02T11:54:18:038386+0000 2024-09-02T11:56:04.922 INFO:journalctl@ceph.mon.b.smithi087.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-02T11:56:04.922 INFO:journalctl@ceph.mon.b.smithi087.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-02T11:56:04.922 INFO:journalctl@ceph.mon.b.smithi087.stdout: legacy client fscid: -1 2024-09-02T11:56:04.922 INFO:journalctl@ceph.mon.b.smithi087.stdout: 2024-09-02T11:56:04.922 INFO:journalctl@ceph.mon.b.smithi087.stdout: No filesystems configured 2024-09-02T11:56:04.922 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-02T11:56:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-02T11:56:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Deploying cephadm binary to smithi087 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:56:05.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Added host smithi087 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi094", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Deploying cephadm binary to smithi094 2024-09-02T11:56:05.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"} : dispatch 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi087:/etc/ceph/ceph.conf 2024-09-02T11:56:05.253 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Added host smithi094 2024-09-02T11:56:05.254 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi094", "name": "osd_memory_target"} : dispatch 2024-09-02T11:56:05.255 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi094:/etc/ceph/ceph.conf 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Deploying cephadm binary to smithi099 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi094:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Added host smithi099 2024-09-02T11:56:05.256 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/4081470598' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/4081470598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.257 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"} : dispatch 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:05.258 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c;count:3 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Deploying daemon mon.c on smithi094 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/1043025568' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.a calling monitor election 2024-09-02T11:56:05.259 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.c calling monitor election 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: monmap epoch 2 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: last_changed 2024-09-02T11:55:56.078299+0000 2024-09-02T11:56:05.260 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: min_mon_release 19 (squid) 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: election_strategy: 1 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: 1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.c 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: fsmap 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mgrmap e13: x(active, since 56s) 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: overall HEALTH_OK 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:05.261 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: Deploying daemon mon.b on smithi087 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/2678507150' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 ceph-mon[21100]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 bash[21082]: e7448cffa948343e4a301e1f20b654f0c8a803e16431427a704f7a1acaefa35e 2024-09-02T11:56:05.262 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:04 smithi087 systemd[1]: Started Ceph mon.b for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T11:56:10.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:56:10.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: mon.a calling monitor election 2024-09-02T11:56:10.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: mon.c calling monitor election 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: mon.b calling monitor election 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: monmap epoch 3 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:10.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: last_changed 2024-09-02T11:56:04.926351+0000 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: min_mon_release 19 (squid) 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: election_strategy: 1 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: 1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.c 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: 2: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.b 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: fsmap 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: mgrmap e13: x(active, since 65s) 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: overall HEALTH_OK 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:56:10.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.286 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: mon.a calling monitor election 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: mon.c calling monitor election 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: mon.b calling monitor election 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: monmap epoch 3 2024-09-02T11:56:10.287 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: last_changed 2024-09-02T11:56:04.926351+0000 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: min_mon_release 19 (squid) 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: election_strategy: 1 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: 1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.c 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: 2: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.b 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: fsmap 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: mgrmap e13: x(active, since 65s) 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: overall HEALTH_OK 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.288 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: mon.a calling monitor election 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: mon.c calling monitor election 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: mon.b calling monitor election 2024-09-02T11:56:10.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: monmap epoch 3 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: fsid e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: last_changed 2024-09-02T11:56:04.926351+0000 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: created 2024-09-02T11:54:14.838508+0000 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: min_mon_release 19 (squid) 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: election_strategy: 1 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: 0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: 1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.c 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: 2: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.b 2024-09-02T11:56:10.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: fsmap 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: mgrmap e13: x(active, since 65s) 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: overall HEALTH_OK 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:10.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:11.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:11.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:11.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:11.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:11.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:11.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:56:11.368 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:56:11.368 INFO:teuthology.orchestra.run.smithi094.stdout:{"epoch":3,"fsid":"e8e22764-6921-11ef-bcd4-c7b262605968","modified":"2024-09-02T11:56:04.926351Z","created":"2024-09-02T11:54:14.838508Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:3300","nonce":0},{"type":"v1","addr":"172.21.15.94:6789","nonce":0}]},"addr":"172.21.15.94:6789/0","public_addr":"172.21.15.94:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-02T11:56:11.368 INFO:teuthology.orchestra.run.smithi094.stderr:dumped monmap epoch 3 2024-09-02T11:56:12.000 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-02T11:56:12.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph config generate-minimal-conf 2024-09-02T11:56:12.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:12.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: Updating smithi064:/etc/ceph/ceph.conf 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: Updating smithi038:/etc/ceph/ceph.conf 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: Updating smithi087:/etc/ceph/ceph.conf 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: Updating smithi094:/etc/ceph/ceph.conf 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/3509166314' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:12.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:12.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:12.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:12.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: Updating smithi064:/etc/ceph/ceph.conf 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: Updating smithi038:/etc/ceph/ceph.conf 2024-09-02T11:56:12.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: Updating smithi087:/etc/ceph/ceph.conf 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: Updating smithi094:/etc/ceph/ceph.conf 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/3509166314' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:12.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:12.486 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:56:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: Updating smithi064:/etc/ceph/ceph.conf 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: Updating smithi038:/etc/ceph/ceph.conf 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: Updating smithi087:/etc/ceph/ceph.conf 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: Updating smithi094:/etc/ceph/ceph.conf 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/3509166314' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:12.502 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:13.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Updating smithi064:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Reconfiguring mon.a (unknown last config time)... 2024-09-02T11:56:13.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:13 smithi064 ceph-mon[17163]: Reconfiguring daemon mon.a on smithi064 2024-09-02T11:56:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Updating smithi064:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Reconfiguring mon.a (unknown last config time)... 2024-09-02T11:56:13.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:13 smithi087 ceph-mon[21100]: Reconfiguring daemon mon.a on smithi064 2024-09-02T11:56:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Updating smithi064:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Updating smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Updating smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Updating smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Updating smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:13.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Reconfiguring mon.a (unknown last config time)... 2024-09-02T11:56:13.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:13 smithi094 ceph-mon[20404]: Reconfiguring daemon mon.a on smithi064 2024-09-02T11:56:13.983 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:13.983 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-09-02T11:56:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T11:56:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:14 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1255166494' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:14.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:14 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1255166494' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:14.583 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-02T11:56:14.584 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:56:14.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:14 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1255166494' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:14.617 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:56:14.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:14.687 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:56:14.687 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T11:56:14.716 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:56:14.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:14.785 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:56:14.785 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T11:56:14.813 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:56:14.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:14.882 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:56:14.882 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T11:56:14.914 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:56:14.915 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:14.982 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:56:14.982 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T11:56:15.014 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:56:15.015 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T11:56:15.116 INFO:tasks.cephadm:Adding mgr.x on smithi064 2024-09-02T11:56:15.116 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch apply mgr '1;smithi064=x' 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: Reconfiguring mon.b (monmap changed)... 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: Reconfiguring daemon mon.b on smithi087 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:15.270 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:15 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:15.355 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:56:15.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: Reconfiguring mon.b (monmap changed)... 2024-09-02T11:56:15.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: Reconfiguring daemon mon.b on smithi087 2024-09-02T11:56:15.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:15.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:15.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:15 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:15.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: Reconfiguring mon.b (monmap changed)... 2024-09-02T11:56:15.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: Reconfiguring daemon mon.b on smithi087 2024-09-02T11:56:15.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:15.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T11:56:15.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T11:56:15.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:15 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:15.977 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled mgr update... 2024-09-02T11:56:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:16 smithi087 ceph-mon[21100]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:16 smithi087 ceph-mon[21100]: Reconfiguring mon.c (monmap changed)... 2024-09-02T11:56:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:16 smithi087 ceph-mon[21100]: Reconfiguring daemon mon.c on smithi094 2024-09-02T11:56:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:16 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:16.520 DEBUG:tasks.cephadm:set 0 configs 2024-09-02T11:56:16.520 INFO:tasks.cephadm:Deploying OSDs... 2024-09-02T11:56:16.520 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:56:16.520 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T11:56:16.537 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T11:56:16.537 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-09-02T11:56:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T11:56:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:16.594 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-02T11:56:16.594 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:16.595 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:16.595 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-02 11:55:28.312385869 +0000 2024-09-02T11:56:16.595 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-02 11:50:01.454843701 +0000 2024-09-02T11:56:16.595 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-02 11:50:01.454843701 +0000 2024-09-02T11:56:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-02 11:50:01.454843701 +0000 2024-09-02T11:56:16.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T11:56:16.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:16 smithi094 ceph-mon[20404]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:16.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:16 smithi094 ceph-mon[20404]: Reconfiguring mon.c (monmap changed)... 2024-09-02T11:56:16.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:16 smithi094 ceph-mon[20404]: Reconfiguring daemon mon.c on smithi094 2024-09-02T11:56:16.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:16 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:16.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:16 smithi064 ceph-mon[17163]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:16.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:16 smithi064 ceph-mon[17163]: Reconfiguring mon.c (monmap changed)... 2024-09-02T11:56:16.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:16 smithi064 ceph-mon[17163]: Reconfiguring daemon mon.c on smithi094 2024-09-02T11:56:16.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:16 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:16.658 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-02T11:56:16.658 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-02T11:56:16.658 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000130872 s, 3.9 MB/s 2024-09-02T11:56:16.660 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T11:56:16.716 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-09-02T11:56:16.771 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-02 11:55:28.336385407 +0000 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-02 11:50:01.793837196 +0000 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-02 11:50:01.793837196 +0000 2024-09-02T11:56:16.772 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-02 11:50:01.793837196 +0000 2024-09-02T11:56:16.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T11:56:16.835 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-02T11:56:16.835 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-02T11:56:16.835 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00026336 s, 1.9 MB/s 2024-09-02T11:56:16.836 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T11:56:16.893 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-02 11:55:28.361384925 +0000 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-02 11:50:02.128830768 +0000 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-02 11:50:02.128830768 +0000 2024-09-02T11:56:16.951 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-02 11:50:02.128830768 +0000 2024-09-02T11:56:16.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T11:56:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-02T11:56:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-02T11:56:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000167843 s, 3.1 MB/s 2024-09-02T11:56:17.015 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T11:56:17.070 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-02 11:55:28.386384443 +0000 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-02 11:50:02.463824340 +0000 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-02 11:50:02.463824340 +0000 2024-09-02T11:56:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-02 11:50:02.463824340 +0000 2024-09-02T11:56:17.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T11:56:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-02T11:56:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-02T11:56:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000264713 s, 1.9 MB/s 2024-09-02T11:56:17.190 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T11:56:17.247 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:56:17.247 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T11:56:17.263 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T11:56:17.263 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-09-02T11:56:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T11:56:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-02 11:55:17.021479169 +0000 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-02 11:50:02.046045857 +0000 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-02 11:50:02.046045857 +0000 2024-09-02T11:56:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-02 11:50:02.046045857 +0000 2024-09-02T11:56:17.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T11:56:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-02T11:56:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-02T11:56:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000247776 s, 2.1 MB/s 2024-09-02T11:56:17.385 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T11:56:17.442 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-09-02T11:56:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T11:56:17.497 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:17.497 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-02T11:56:17.498 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:17.498 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:17.498 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-02 11:55:17.054478527 +0000 2024-09-02T11:56:17.498 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-02 11:50:02.374039419 +0000 2024-09-02T11:56:17.498 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-02 11:50:02.374039419 +0000 2024-09-02T11:56:17.498 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-02 11:50:02.374039419 +0000 2024-09-02T11:56:17.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T11:56:17.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi064=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:17.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: Saving service mgr spec with placement smithi064=x;count:1 2024-09-02T11:56:17.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T11:56:17.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:17.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi064=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:17.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: Saving service mgr spec with placement smithi064=x;count:1 2024-09-02T11:56:17.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T11:56:17.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-02T11:56:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-02T11:56:17.561 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.00021563 s, 2.4 MB/s 2024-09-02T11:56:17.562 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T11:56:17.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi064=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:17.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: Saving service mgr spec with placement smithi064=x;count:1 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T11:56:17.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:17.617 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-09-02T11:56:17.673 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T11:56:17.673 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:17.673 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-02T11:56:17.673 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:17.673 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:17.673 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-02 11:55:17.085477923 +0000 2024-09-02T11:56:17.674 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-02 11:50:02.703032961 +0000 2024-09-02T11:56:17.674 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-02 11:50:02.703032961 +0000 2024-09-02T11:56:17.674 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-02 11:50:02.703032961 +0000 2024-09-02T11:56:17.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T11:56:17.736 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-02T11:56:17.736 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-02T11:56:17.736 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000235641 s, 2.2 MB/s 2024-09-02T11:56:17.737 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T11:56:17.793 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-02 11:55:17.120477242 +0000 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-02 11:50:03.014026857 +0000 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-02 11:50:03.014026857 +0000 2024-09-02T11:56:17.850 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-02 11:50:03.014026857 +0000 2024-09-02T11:56:17.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T11:56:17.913 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-02T11:56:17.913 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-02T11:56:17.913 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000196743 s, 2.6 MB/s 2024-09-02T11:56:17.915 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T11:56:17.971 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:56:17.971 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T11:56:17.990 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T11:56:17.990 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-09-02T11:56:18.047 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-02 11:56:04.378085773 +0000 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-02 11:50:01.632083363 +0000 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-02 11:50:01.632083363 +0000 2024-09-02T11:56:18.048 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-02 11:50:01.632083363 +0000 2024-09-02T11:56:18.048 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T11:56:18.116 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-02T11:56:18.116 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-02T11:56:18.116 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000286579 s, 1.8 MB/s 2024-09-02T11:56:18.117 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T11:56:18.176 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-09-02T11:56:18.233 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T11:56:18.233 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:18.233 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-02T11:56:18.233 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:18.233 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:18.234 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-02 11:56:04.378085773 +0000 2024-09-02T11:56:18.234 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-02 11:50:01.958073797 +0000 2024-09-02T11:56:18.234 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-02 11:50:01.958073797 +0000 2024-09-02T11:56:18.234 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-02 11:50:01.958073797 +0000 2024-09-02T11:56:18.234 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T11:56:18.299 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-02T11:56:18.299 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-02T11:56:18.299 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000214697 s, 2.4 MB/s 2024-09-02T11:56:18.300 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T11:56:18.358 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-09-02T11:56:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T11:56:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:18.414 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-02T11:56:18.414 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:18.414 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:18.415 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-02 11:56:04.378085773 +0000 2024-09-02T11:56:18.415 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-02 11:50:02.279064377 +0000 2024-09-02T11:56:18.415 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-02 11:50:02.279064377 +0000 2024-09-02T11:56:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-02 11:50:02.279064377 +0000 2024-09-02T11:56:18.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T11:56:18.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:18 smithi087 ceph-mon[21100]: Reconfiguring mgr.x (unknown last config time)... 2024-09-02T11:56:18.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:18 smithi087 ceph-mon[21100]: Reconfiguring daemon mgr.x on smithi064 2024-09-02T11:56:18.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:18 smithi087 ceph-mon[21100]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:18.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:18 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:18.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:18 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:18.478 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-02T11:56:18.478 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-02T11:56:18.478 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00023523 s, 2.2 MB/s 2024-09-02T11:56:18.479 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T11:56:18.537 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-02 11:56:04.378085773 +0000 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-02 11:50:02.624054253 +0000 2024-09-02T11:56:18.593 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-02 11:50:02.624054253 +0000 2024-09-02T11:56:18.594 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-02 11:50:02.624054253 +0000 2024-09-02T11:56:18.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T11:56:18.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:18 smithi064 ceph-mon[17163]: Reconfiguring mgr.x (unknown last config time)... 2024-09-02T11:56:18.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:18 smithi064 ceph-mon[17163]: Reconfiguring daemon mgr.x on smithi064 2024-09-02T11:56:18.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:18 smithi064 ceph-mon[17163]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:18.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:18.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:18 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:18 smithi094 ceph-mon[20404]: Reconfiguring mgr.x (unknown last config time)... 2024-09-02T11:56:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:18 smithi094 ceph-mon[20404]: Reconfiguring daemon mgr.x on smithi064 2024-09-02T11:56:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:18 smithi094 ceph-mon[20404]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:18 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:18 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-02T11:56:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-02T11:56:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000197719 s, 2.6 MB/s 2024-09-02T11:56:18.658 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T11:56:18.714 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:56:18.714 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T11:56:18.734 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T11:56:18.734 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-02 11:55:55.510489149 +0000 2024-09-02T11:56:18.793 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-02 11:50:03.972877918 +0000 2024-09-02T11:56:18.794 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-02 11:50:03.972877918 +0000 2024-09-02T11:56:18.794 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-02 11:50:03.972877918 +0000 2024-09-02T11:56:18.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T11:56:18.863 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-02T11:56:18.863 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-02T11:56:18.863 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000132292 s, 3.9 MB/s 2024-09-02T11:56:18.864 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T11:56:18.923 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-02 11:55:55.511489134 +0000 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-02 11:50:04.288873722 +0000 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-02 11:50:04.288873722 +0000 2024-09-02T11:56:18.981 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-02 11:50:04.288873722 +0000 2024-09-02T11:56:18.982 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T11:56:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-02T11:56:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-02T11:56:19.047 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000101995 s, 5.0 MB/s 2024-09-02T11:56:19.048 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T11:56:19.106 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2024-09-02T11:56:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T11:56:19.162 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:19.162 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-09-02T11:56:19.162 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:19.162 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:19.163 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-02 11:55:55.511489134 +0000 2024-09-02T11:56:19.163 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-02 11:50:04.602869551 +0000 2024-09-02T11:56:19.163 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-02 11:50:04.602869551 +0000 2024-09-02T11:56:19.163 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-02 11:50:04.602869551 +0000 2024-09-02T11:56:19.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T11:56:19.227 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-02T11:56:19.227 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-02T11:56:19.227 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 9.0676e-05 s, 5.6 MB/s 2024-09-02T11:56:19.228 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T11:56:19.285 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2024-09-02T11:56:19.341 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T11:56:19.341 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:19.341 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-02T11:56:19.341 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:19.342 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:19.342 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2024-09-02 11:55:55.511489134 +0000 2024-09-02T11:56:19.342 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2024-09-02 11:50:04.903865554 +0000 2024-09-02T11:56:19.342 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2024-09-02 11:50:04.903865554 +0000 2024-09-02T11:56:19.342 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: 2024-09-02 11:50:04.903865554 +0000 2024-09-02T11:56:19.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T11:56:19.404 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2024-09-02T11:56:19.404 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2024-09-02T11:56:19.405 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000109258 s, 4.7 MB/s 2024-09-02T11:56:19.406 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T11:56:19.462 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:56:19.462 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T11:56:19.478 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T11:56:19.479 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-09-02T11:56:19.534 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-02 11:55:49.542680911 +0000 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-02 11:50:03.845008272 +0000 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-02 11:50:03.845008272 +0000 2024-09-02T11:56:19.535 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-02 11:50:03.845008272 +0000 2024-09-02T11:56:19.535 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T11:56:19.599 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-02T11:56:19.599 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-02T11:56:19.599 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000261455 s, 2.0 MB/s 2024-09-02T11:56:19.600 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T11:56:19.657 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-02 11:55:49.568680278 +0000 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-02 11:50:04.172998724 +0000 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-02 11:50:04.172998724 +0000 2024-09-02T11:56:19.713 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-02 11:50:04.172998724 +0000 2024-09-02T11:56:19.714 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T11:56:19.776 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-02T11:56:19.776 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-02T11:56:19.776 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000221881 s, 2.3 MB/s 2024-09-02T11:56:19.777 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T11:56:19.834 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-09-02T11:56:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T11:56:19.888 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-02 11:55:49.593679668 +0000 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-02 11:50:04.512988828 +0000 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-02 11:50:04.512988828 +0000 2024-09-02T11:56:19.889 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-02 11:50:04.512988828 +0000 2024-09-02T11:56:19.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T11:56:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-02T11:56:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-02T11:56:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000207652 s, 2.5 MB/s 2024-09-02T11:56:19.953 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T11:56:20.010 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-02 11:55:49.619679034 +0000 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-02 11:50:04.837979368 +0000 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-02 11:50:04.837979368 +0000 2024-09-02T11:56:20.065 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-02 11:50:04.837979368 +0000 2024-09-02T11:56:20.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T11:56:20.127 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-02T11:56:20.128 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-02T11:56:20.128 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000250834 s, 2.0 MB/s 2024-09-02T11:56:20.129 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T11:56:20.185 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/vg_nvme/lv_4... 2024-09-02T11:56:20.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-02T11:56:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:56:20.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:20 smithi064 ceph-mon[17163]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:20.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:20 smithi094 ceph-mon[20404]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:20.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:20 smithi087 ceph-mon[21100]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:22.493 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:22 smithi064 ceph-mon[17163]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:22.552 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:22 smithi087 ceph-mon[21100]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:22 smithi094 ceph-mon[20404]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:56:22.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-09-02T11:56:23.075 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:56:24.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:24 smithi064 ceph-mon[17163]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:24.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:56:24.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:56:24.387 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:24 smithi094 ceph-mon[20404]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:56:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:56:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:24 smithi087 ceph-mon[21100]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:56:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:56:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:25.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:25 smithi064 ceph-mon[17163]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:25.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:25 smithi094 ceph-mon[20404]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:25.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:25 smithi087 ceph-mon[21100]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:26.577 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:26 smithi064 ceph-mon[17163]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:26.577 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:26 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/346252673' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "25b94b5e-a25e-49ac-ab44-e73f48429152"} : dispatch 2024-09-02T11:56:26.577 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:26 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/346252673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25b94b5e-a25e-49ac-ab44-e73f48429152"}]': finished 2024-09-02T11:56:26.577 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:26 smithi064 ceph-mon[17163]: osdmap e5: 1 total, 0 up, 1 in 2024-09-02T11:56:26.577 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:26.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:26 smithi094 ceph-mon[20404]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:26.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:26 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/346252673' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "25b94b5e-a25e-49ac-ab44-e73f48429152"} : dispatch 2024-09-02T11:56:26.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:26 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/346252673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25b94b5e-a25e-49ac-ab44-e73f48429152"}]': finished 2024-09-02T11:56:26.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:26 smithi094 ceph-mon[20404]: osdmap e5: 1 total, 0 up, 1 in 2024-09-02T11:56:26.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:26 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:26 smithi087 ceph-mon[21100]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:26 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/346252673' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "25b94b5e-a25e-49ac-ab44-e73f48429152"} : dispatch 2024-09-02T11:56:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:26 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/346252673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25b94b5e-a25e-49ac-ab44-e73f48429152"}]': finished 2024-09-02T11:56:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:26 smithi087 ceph-mon[21100]: osdmap e5: 1 total, 0 up, 1 in 2024-09-02T11:56:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:26 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:27.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:27 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3241194785' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:56:27.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:27 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3241194785' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:56:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:27 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3241194785' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:56:28.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:28 smithi064 ceph-mon[17163]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:28.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:28 smithi094 ceph-mon[20404]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:28.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:28 smithi087 ceph-mon[21100]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:30.498 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:30 smithi064 ceph-mon[17163]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:30 smithi094 ceph-mon[20404]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:30.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:30 smithi087 ceph-mon[21100]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:32.469 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:32 smithi064 ceph-mon[17163]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:32.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:32 smithi094 ceph-mon[20404]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:32.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:32 smithi087 ceph-mon[21100]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-02T11:56:33.552 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:33.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:33 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-02T11:56:33.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:33 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:33.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:33 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-02T11:56:33.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:33 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:34.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:34 smithi064 ceph-mon[17163]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:34.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:34 smithi064 ceph-mon[17163]: Deploying daemon osd.0 on smithi064 2024-09-02T11:56:34.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:34 smithi094 ceph-mon[20404]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:34.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:34 smithi094 ceph-mon[20404]: Deploying daemon osd.0 on smithi064 2024-09-02T11:56:34.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:34 smithi087 ceph-mon[21100]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:34.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:34 smithi087 ceph-mon[21100]: Deploying daemon osd.0 on smithi064 2024-09-02T11:56:36.466 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:36 smithi064 ceph-mon[17163]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:36.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:36 smithi094 ceph-mon[20404]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:36.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:36 smithi087 ceph-mon[21100]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:38.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:38 smithi064 ceph-mon[17163]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:38.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:38 smithi094 ceph-mon[20404]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:38.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:38 smithi087 ceph-mon[21100]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:39.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:39.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:39.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:39.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:39.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:39.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:40.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:40 smithi064 ceph-mon[17163]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:40.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:40 smithi094 ceph-mon[20404]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:40.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:40 smithi087 ceph-mon[21100]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:42.155 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2024-09-02T11:56:42.393 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:42 smithi064 ceph-mon[17163]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:42.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:42.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:42.394 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:42 smithi087 ceph-mon[21100]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:42.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:42 smithi094 ceph-mon[20404]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:42.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:42.781 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0.service 2024-09-02T11:56:42.784 INFO:tasks.cephadm:Deploying osd.1 on smithi064 with /dev/vg_nvme/lv_3... 2024-09-02T11:56:42.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-02T11:56:43.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:43 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:43.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:43 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:43.316 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:43 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:43.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:43 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:43.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:43 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:43.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:43 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:43.594 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:56:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0[28917]: 2024-09-02T11:56:43.456+0000 7fc2a9338640 -1 osd.0 0 log_to_monitors true 2024-09-02T11:56:43.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:43 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:56:43.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:43 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:43.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:43 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:43.925 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:56:44.195 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:44 smithi064 ceph-mon[17163]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:44.195 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:44 smithi064 ceph-mon[17163]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-02T11:56:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:44 smithi087 ceph-mon[21100]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:44 smithi087 ceph-mon[21100]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-02T11:56:44.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:44 smithi094 ceph-mon[20404]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:44.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:44 smithi094 ceph-mon[20404]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-02T11:56:45.344 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0[28917]: 2024-09-02T11:56:45.158+0000 7fc2a52bb640 -1 osd.0 0 waiting for initial osdmap 2024-09-02T11:56:45.344 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0[28917]: 2024-09-02T11:56:45.164+0000 7fc2a10e2640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:56:45.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-mon[17163]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-02T11:56:45.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-mon[17163]: osdmap e6: 1 total, 0 up, 1 in 2024-09-02T11:56:45.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-mon[17163]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]} : dispatch 2024-09-02T11:56:45.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:45.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:45 smithi064 ceph-mon[17163]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-09-02T11:56:45.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:45 smithi087 ceph-mon[21100]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-02T11:56:45.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:45 smithi087 ceph-mon[21100]: osdmap e6: 1 total, 0 up, 1 in 2024-09-02T11:56:45.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:45 smithi087 ceph-mon[21100]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]} : dispatch 2024-09-02T11:56:45.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:45 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:45.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:45 smithi087 ceph-mon[21100]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-09-02T11:56:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:45 smithi094 ceph-mon[20404]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-02T11:56:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:45 smithi094 ceph-mon[20404]: osdmap e6: 1 total, 0 up, 1 in 2024-09-02T11:56:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:45 smithi094 ceph-mon[20404]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]} : dispatch 2024-09-02T11:56:45.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:45 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:45.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:45 smithi094 ceph-mon[20404]: from='osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-09-02T11:56:46.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:46 smithi087 ceph-mon[21100]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:46.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:46 smithi087 ceph-mon[21100]: osdmap e7: 1 total, 0 up, 1 in 2024-09-02T11:56:46.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:46 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:46 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:46 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:46 smithi064 ceph-mon[17163]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:46.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:46 smithi064 ceph-mon[17163]: osdmap e7: 1 total, 0 up, 1 in 2024-09-02T11:56:46.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:46 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:46 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:46 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:46 smithi094 ceph-mon[20404]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:46 smithi094 ceph-mon[20404]: osdmap e7: 1 total, 0 up, 1 in 2024-09-02T11:56:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:46 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:46 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:46 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:47.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:56:47.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:56:47.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183] boot 2024-09-02T11:56:47.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: osdmap e8: 1 total, 1 up, 1 in 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi064 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi064 to 17088M 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:47.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:56:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:56:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183] boot 2024-09-02T11:56:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: osdmap e8: 1 total, 1 up, 1 in 2024-09-02T11:56:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:47.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi064 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi064 to 17088M 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:47.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.567 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:56:47.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-09-02T11:56:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: osd.0 [v2:172.21.15.64:6802/2163384183,v1:172.21.15.64:6803/2163384183] boot 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: osdmap e8: 1 total, 1 up, 1 in 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi064 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi064 to 17088M 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:47.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:56:47.598 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:56:47.840 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:56:48.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:48 smithi087 ceph-mon[21100]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:48.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:48 smithi087 ceph-mon[21100]: osdmap e9: 1 total, 1 up, 1 in 2024-09-02T11:56:48.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:48 smithi064 ceph-mon[17163]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:48.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:48 smithi064 ceph-mon[17163]: osdmap e9: 1 total, 1 up, 1 in 2024-09-02T11:56:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:48 smithi094 ceph-mon[20404]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T11:56:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:48 smithi094 ceph-mon[20404]: osdmap e9: 1 total, 1 up, 1 in 2024-09-02T11:56:49.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:49 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:56:49.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:49 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:56:49.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:49 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:49 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:56:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:49 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:56:49.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:49 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:49.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:49 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:56:49.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:49 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:56:49.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:49 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:50.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:50 smithi064 ceph-mon[17163]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:50.461 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:50 smithi064 ceph-mon[17163]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:50.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:50 smithi087 ceph-mon[21100]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:50.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:50 smithi087 ceph-mon[21100]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:50.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:50 smithi094 ceph-mon[20404]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:50.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:50 smithi094 ceph-mon[20404]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:56:51.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:51 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1725678677' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dd005fa8-f0f3-497a-a20c-b1706bf3c3de"} : dispatch 2024-09-02T11:56:51.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:51 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1725678677' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd005fa8-f0f3-497a-a20c-b1706bf3c3de"}]': finished 2024-09-02T11:56:51.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:51 smithi064 ceph-mon[17163]: osdmap e10: 2 total, 1 up, 2 in 2024-09-02T11:56:51.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:51 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:56:51.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:51 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1725678677' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dd005fa8-f0f3-497a-a20c-b1706bf3c3de"} : dispatch 2024-09-02T11:56:51.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:51 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1725678677' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd005fa8-f0f3-497a-a20c-b1706bf3c3de"}]': finished 2024-09-02T11:56:51.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:51 smithi094 ceph-mon[20404]: osdmap e10: 2 total, 1 up, 2 in 2024-09-02T11:56:51.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:51 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:56:51.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:51 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1725678677' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dd005fa8-f0f3-497a-a20c-b1706bf3c3de"} : dispatch 2024-09-02T11:56:51.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:51 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1725678677' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd005fa8-f0f3-497a-a20c-b1706bf3c3de"}]': finished 2024-09-02T11:56:51.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:51 smithi087 ceph-mon[21100]: osdmap e10: 2 total, 1 up, 2 in 2024-09-02T11:56:51.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:51 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:56:52.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:52 smithi064 ceph-mon[17163]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:52.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:52 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3054087743' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:56:52.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:52 smithi094 ceph-mon[20404]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:52.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:52 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3054087743' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:56:52.651 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:52 smithi087 ceph-mon[21100]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:52.651 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:52 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3054087743' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:56:54.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:54 smithi064 ceph-mon[17163]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:54.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:54 smithi094 ceph-mon[20404]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:54.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:54 smithi087 ceph-mon[21100]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:56.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:56 smithi064 ceph-mon[17163]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:56.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:56 smithi094 ceph-mon[20404]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:56.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:56 smithi087 ceph-mon[21100]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:58.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:58 smithi064 ceph-mon[17163]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:58.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:58 smithi094 ceph-mon[20404]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:58.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:58 smithi087 ceph-mon[21100]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:56:59.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:59 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-02T11:56:59.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:56:59 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:59.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:59 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-02T11:56:59.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:56:59 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:56:59.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:59 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-02T11:56:59.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:56:59 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:00.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:00 smithi064 ceph-mon[17163]: Deploying daemon osd.1 on smithi064 2024-09-02T11:57:00.356 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:00 smithi064 ceph-mon[17163]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:00.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:00 smithi094 ceph-mon[20404]: Deploying daemon osd.1 on smithi064 2024-09-02T11:57:00.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:00 smithi094 ceph-mon[20404]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:00.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:00 smithi087 ceph-mon[21100]: Deploying daemon osd.1 on smithi064 2024-09-02T11:57:00.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:00 smithi087 ceph-mon[21100]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:02.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:02 smithi064 ceph-mon[17163]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:02.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:02 smithi094 ceph-mon[20404]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:02 smithi087 ceph-mon[21100]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:04.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:04 smithi064 ceph-mon[17163]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:04.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:04.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:04.460 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:04.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:04 smithi094 ceph-mon[20404]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:04.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:04.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:04.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:04.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:04 smithi087 ceph-mon[21100]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:04.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:04.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:04.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:06 smithi064 ceph-mon[17163]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:06 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:06 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:06 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:06.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:06 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:06.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:06 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:06 smithi094 ceph-mon[20404]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:06.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:06 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:06 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:06 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:06.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:06 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:06.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:06 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:06 smithi087 ceph-mon[21100]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:06 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:06 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:06 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:06 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:06 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:07.638 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 1 on host 'smithi064' 2024-09-02T11:57:08.244 DEBUG:teuthology.orchestra.run.smithi064:osd.1> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1.service 2024-09-02T11:57:08.246 INFO:tasks.cephadm:Deploying osd.2 on smithi087 with /dev/vg_nvme/lv_4... 2024-09-02T11:57:08.247 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-02T11:57:08.473 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:08 smithi064 ceph-mon[17163]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:08.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:08.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:08.474 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:08.494 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:57:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:08 smithi094 ceph-mon[20404]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:08.615 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:08 smithi087 ceph-mon[21100]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:08.615 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:08.615 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:08.615 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:08.729 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1[34812]: 2024-09-02T11:57:08.478+0000 7fb4ecca8640 -1 osd.1 0 log_to_monitors true 2024-09-02T11:57:09.449 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:09 smithi064 ceph-mon[17163]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-02T11:57:09.562 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:09 smithi087 ceph-mon[21100]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-02T11:57:09.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:09 smithi094 ceph-mon[20404]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-02T11:57:10.398 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-mon[17163]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:10.399 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-mon[17163]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-02T11:57:10.399 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-mon[17163]: osdmap e11: 2 total, 1 up, 2 in 2024-09-02T11:57:10.399 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-mon[17163]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]} : dispatch 2024-09-02T11:57:10.399 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:10.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:10 smithi094 ceph-mon[20404]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:10.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:10 smithi094 ceph-mon[20404]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-02T11:57:10.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:10 smithi094 ceph-mon[20404]: osdmap e11: 2 total, 1 up, 2 in 2024-09-02T11:57:10.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:10 smithi094 ceph-mon[20404]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]} : dispatch 2024-09-02T11:57:10.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:10 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:10.643 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:10 smithi087 ceph-mon[21100]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:10.643 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:10 smithi087 ceph-mon[21100]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-02T11:57:10.643 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:10 smithi087 ceph-mon[21100]: osdmap e11: 2 total, 1 up, 2 in 2024-09-02T11:57:10.644 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:10 smithi087 ceph-mon[21100]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]} : dispatch 2024-09-02T11:57:10.644 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:10 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:10.734 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:57:10.761 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_4 2024-09-02T11:57:11.014 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:57:11.094 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1[34812]: 2024-09-02T11:57:10.733+0000 7fb4e8c2b640 -1 osd.1 0 waiting for initial osdmap 2024-09-02T11:57:11.094 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1[34812]: 2024-09-02T11:57:10.736+0000 7fb4e0a4a640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:57:11.430 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-09-02T11:57:11.430 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: osdmap e12: 2 total, 1 up, 2 in 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.431 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.438 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: osdmap e12: 2 total, 1 up, 2 in 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.439 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-09-02T11:57:11.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: osdmap e12: 2 total, 1 up, 2 in 2024-09-02T11:57:11.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301]' entity='osd.1' 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:11.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:12.421 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:57:12.421 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:57:12.421 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:12.421 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi064 2024-09-02T11:57:12.421 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi064 to 8544M 2024-09-02T11:57:12.422 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:57:12.422 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:57:12.422 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:12.422 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301] boot 2024-09-02T11:57:12.422 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: osdmap e13: 2 total, 2 up, 2 in 2024-09-02T11:57:12.422 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:57:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:57:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi064 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi064 to 8544M 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301] boot 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: osdmap e13: 2 total, 2 up, 2 in 2024-09-02T11:57:12.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:57:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:57:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T11:57:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi064 2024-09-02T11:57:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi064 to 8544M 2024-09-02T11:57:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:57:12.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:57:12.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:12.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: osd.1 [v2:172.21.15.64:6810/1982412301,v1:172.21.15.64:6811/1982412301] boot 2024-09-02T11:57:12.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: osdmap e13: 2 total, 2 up, 2 in 2024-09-02T11:57:12.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T11:57:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:13 smithi087 ceph-mon[21100]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:57:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:13 smithi064 ceph-mon[17163]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:57:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:13 smithi094 ceph-mon[20404]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:57:14.456 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:14.457 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: osdmap e14: 2 total, 2 up, 2 in 2024-09-02T11:57:14.457 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/4036131460' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"} : dispatch 2024-09-02T11:57:14.457 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"} : dispatch 2024-09-02T11:57:14.457 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"}]': finished 2024-09-02T11:57:14.457 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: osdmap e15: 3 total, 2 up, 3 in 2024-09-02T11:57:14.457 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: osdmap e14: 2 total, 2 up, 2 in 2024-09-02T11:57:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/4036131460' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"} : dispatch 2024-09-02T11:57:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"} : dispatch 2024-09-02T11:57:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"}]': finished 2024-09-02T11:57:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: osdmap e15: 3 total, 2 up, 3 in 2024-09-02T11:57:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: osdmap e14: 2 total, 2 up, 2 in 2024-09-02T11:57:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/4036131460' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"} : dispatch 2024-09-02T11:57:14.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"} : dispatch 2024-09-02T11:57:14.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9a8efd8-9fae-4319-aa01-801cba9780a5"}]': finished 2024-09-02T11:57:14.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: osdmap e15: 3 total, 2 up, 3 in 2024-09-02T11:57:14.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:15.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:15 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/1996665498' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:57:15.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:15 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/1996665498' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:57:15.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:15 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/1996665498' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:57:16.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:16 smithi064 ceph-mon[17163]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:16.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:16 smithi094 ceph-mon[20404]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:16.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:16 smithi087 ceph-mon[21100]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:18.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:18 smithi064 ceph-mon[17163]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:18 smithi094 ceph-mon[20404]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:18.607 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:18 smithi087 ceph-mon[21100]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:20.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:20 smithi087 ceph-mon[21100]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:20.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:20 smithi064 ceph-mon[17163]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:20.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:20 smithi094 ceph-mon[20404]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:21.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:21 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-02T11:57:21.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:21 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:21.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:21 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-02T11:57:21.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:21 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:21.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:21 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-02T11:57:21.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:21 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:22.517 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:22 smithi087 ceph-mon[21100]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:22.517 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:22 smithi087 ceph-mon[21100]: Deploying daemon osd.2 on smithi087 2024-09-02T11:57:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:22 smithi064 ceph-mon[17163]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:22 smithi064 ceph-mon[17163]: Deploying daemon osd.2 on smithi087 2024-09-02T11:57:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:22 smithi094 ceph-mon[20404]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:22 smithi094 ceph-mon[20404]: Deploying daemon osd.2 on smithi087 2024-09-02T11:57:23.386 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:23 smithi087 ceph-mon[21100]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:23.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:23 smithi064 ceph-mon[17163]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:23.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:23 smithi094 ceph-mon[20404]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:25.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:25 smithi064 ceph-mon[17163]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:25.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:25 smithi094 ceph-mon[20404]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:25.933 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:25 smithi087 ceph-mon[21100]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:26 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:26 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:26 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:26.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:26.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:26.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:26.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:26 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:26.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:26 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:26.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:26 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:27.825 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:27 smithi087 ceph-mon[21100]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:27.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:27 smithi064 ceph-mon[17163]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:27.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:27 smithi094 ceph-mon[20404]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:30.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:29 smithi087 ceph-mon[21100]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:30.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:30.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:30.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:29 smithi064 ceph-mon[17163]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:30.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:30.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:30.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:29 smithi094 ceph-mon[20404]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.549 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 2 on host 'smithi087' 2024-09-02T11:57:30.898 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:30.898 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:30.898 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:31.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:31.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:31.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:31.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:31.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:31.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:31.237 DEBUG:teuthology.orchestra.run.smithi087:osd.2> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2.service 2024-09-02T11:57:31.239 INFO:tasks.cephadm:Deploying osd.3 on smithi087 with /dev/vg_nvme/lv_3... 2024-09-02T11:57:31.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-02T11:57:32.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:31 smithi087 ceph-mon[21100]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:32.000 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:31 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2[24647]: 2024-09-02T11:57:31.765+0000 7ff985841640 -1 osd.2 0 log_to_monitors true 2024-09-02T11:57:32.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:31 smithi094 ceph-mon[20404]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:32.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:31 smithi064 ceph-mon[17163]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:32.822 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:57:32.943 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:32 smithi087 ceph-mon[21100]: from='osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T11:57:32.944 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:32 smithi087 ceph-mon[21100]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T11:57:33.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:32 smithi064 ceph-mon[17163]: from='osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T11:57:33.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:32 smithi064 ceph-mon[17163]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T11:57:33.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:32 smithi094 ceph-mon[20404]: from='osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T11:57:33.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:32 smithi094 ceph-mon[20404]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T11:57:34.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:33 smithi087 ceph-mon[21100]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:34.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:33 smithi087 ceph-mon[21100]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-02T11:57:34.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:33 smithi087 ceph-mon[21100]: osdmap e16: 3 total, 2 up, 3 in 2024-09-02T11:57:34.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:33 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:34.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:33 smithi087 ceph-mon[21100]: from='osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:34.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:33 smithi087 ceph-mon[21100]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:34.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:33 smithi064 ceph-mon[17163]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:34.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:33 smithi064 ceph-mon[17163]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-02T11:57:34.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:33 smithi064 ceph-mon[17163]: osdmap e16: 3 total, 2 up, 3 in 2024-09-02T11:57:34.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:34.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:33 smithi064 ceph-mon[17163]: from='osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:34.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:33 smithi064 ceph-mon[17163]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:34.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:33 smithi094 ceph-mon[20404]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:34.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:33 smithi094 ceph-mon[20404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-02T11:57:34.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:33 smithi094 ceph-mon[20404]: osdmap e16: 3 total, 2 up, 3 in 2024-09-02T11:57:34.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:33 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:34.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:33 smithi094 ceph-mon[20404]: from='osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:34.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:33 smithi094 ceph-mon[20404]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:34.419 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2[24647]: 2024-09-02T11:57:34.186+0000 7ff9817c4640 -1 osd.2 0 waiting for initial osdmap 2024-09-02T11:57:34.420 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2[24647]: 2024-09-02T11:57:34.189+0000 7ff9795e3640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:57:35.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:57:35.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: osdmap e17: 3 total, 2 up, 3 in 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi087 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi087 to 21184M 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:34 smithi087 ceph-mon[21100]: from='osd.2 ' entity='osd.2' 2024-09-02T11:57:35.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:57:35.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: osdmap e17: 3 total, 2 up, 3 in 2024-09-02T11:57:35.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi087 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi087 to 21184M 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:34 smithi064 ceph-mon[17163]: from='osd.2 ' entity='osd.2' 2024-09-02T11:57:35.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:57:35.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: osdmap e17: 3 total, 2 up, 3 in 2024-09-02T11:57:35.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi087 2024-09-02T11:57:35.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi087 to 21184M 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:35.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:34 smithi094 ceph-mon[20404]: from='osd.2 ' entity='osd.2' 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796] boot 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: osdmap e18: 3 total, 3 up, 3 in 2024-09-02T11:57:35.985 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:35 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:35.987 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:57:36.012 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_3 2024-09-02T11:57:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:57:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:57:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796] boot 2024-09-02T11:57:36.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: osdmap e18: 3 total, 3 up, 3 in 2024-09-02T11:57:36.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:35 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: osd.2 [v2:172.21.15.87:6800/3018291796,v1:172.21.15.87:6801/3018291796] boot 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: osdmap e18: 3 total, 3 up, 3 in 2024-09-02T11:57:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:35 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T11:57:36.253 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:57:38.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: pgmap v84: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:38.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: osdmap e19: 3 total, 3 up, 3 in 2024-09-02T11:57:38.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:57:38.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:57:38.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:57:38.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:38.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-02T11:57:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: pgmap v84: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: osdmap e19: 3 total, 3 up, 3 in 2024-09-02T11:57:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:57:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:57:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:57:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-02T11:57:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: pgmap v84: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: osdmap e19: 3 total, 3 up, 3 in 2024-09-02T11:57:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:57:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:57:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:57:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:38.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-02T11:57:39.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:38 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-02T11:57:39.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:38 smithi064 ceph-mon[17163]: osdmap e20: 3 total, 3 up, 3 in 2024-09-02T11:57:39.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:38 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-02T11:57:39.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:38 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-02T11:57:39.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:38 smithi094 ceph-mon[20404]: osdmap e20: 3 total, 3 up, 3 in 2024-09-02T11:57:39.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:38 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-02T11:57:39.117 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:38 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-02T11:57:39.117 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:38 smithi087 ceph-mon[21100]: osdmap e20: 3 total, 3 up, 3 in 2024-09-02T11:57:39.117 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:38 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-02T11:57:39.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38306]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38306]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38306]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38306]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38309]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38309]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38309]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38309]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.345 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38294]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38294]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38294]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38294]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38297]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38297]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38297]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38297]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38300]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38300]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.346 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38300]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.347 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38303]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T11:57:39.347 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38303]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.347 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38303]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.347 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 11:57:39 smithi064 sudo[38303]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.474 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28852]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T11:57:39.474 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28852]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.475 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28852]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.475 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28852]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.475 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28855]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T11:57:39.475 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28855]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.475 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28855]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.475 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28855]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T11:57:39.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28864]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28864]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28867]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T11:57:39.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28867]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28867]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 sudo[28867]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22004]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T11:57:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22004]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22004]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:39.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22007]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T11:57:39.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T11:57:39.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22007]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T11:57:39.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 sudo[22007]: pam_unix(sudo:session): session closed for user root 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: osdmap e21: 3 total, 3 up, 3 in 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/4035942600' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"}]': finished 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: osdmap e22: 4 total, 3 up, 4 in 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:39 smithi064 ceph-mon[17163]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:40.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: osdmap e21: 3 total, 3 up, 3 in 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/4035942600' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"} : dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"} : dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"}]': finished 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: osdmap e22: 4 total, 3 up, 4 in 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.100 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:39 smithi094 ceph-mon[20404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:40.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-02T11:57:40.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: osdmap e21: 3 total, 3 up, 3 in 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/4035942600' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"} : dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"} : dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59c9bbd9-a906-46a3-a870-40e7fad00805"}]': finished 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: osdmap e22: 4 total, 3 up, 4 in 2024-09-02T11:57:40.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T11:57:40.252 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:39 smithi087 ceph-mon[21100]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T11:57:41.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:40 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/467041231' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:57:41.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:40 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/467041231' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:57:41.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:40 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/467041231' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:57:42.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:41 smithi064 ceph-mon[17163]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:42.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:41 smithi064 ceph-mon[17163]: mgrmap e14: x(active, since 2m) 2024-09-02T11:57:42.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:41 smithi094 ceph-mon[20404]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:42.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:41 smithi094 ceph-mon[20404]: mgrmap e14: x(active, since 2m) 2024-09-02T11:57:42.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:41 smithi087 ceph-mon[21100]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:42.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:41 smithi087 ceph-mon[21100]: mgrmap e14: x(active, since 2m) 2024-09-02T11:57:44.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:43 smithi087 ceph-mon[21100]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:44.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:43 smithi064 ceph-mon[17163]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:44.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:43 smithi094 ceph-mon[20404]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:46.073 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:45 smithi087 ceph-mon[21100]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:46.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:45 smithi064 ceph-mon[17163]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:46.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:45 smithi094 ceph-mon[20404]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:47 smithi064 ceph-mon[17163]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-02T11:57:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:47 smithi064 ceph-mon[17163]: Deploying daemon osd.3 on smithi087 2024-09-02T11:57:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:47 smithi094 ceph-mon[20404]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-02T11:57:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:47 smithi094 ceph-mon[20404]: Deploying daemon osd.3 on smithi087 2024-09-02T11:57:48.138 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:47 smithi087 ceph-mon[21100]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:48.138 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-02T11:57:48.138 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:48.138 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:47 smithi087 ceph-mon[21100]: Deploying daemon osd.3 on smithi087 2024-09-02T11:57:49.955 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:49 smithi087 ceph-mon[21100]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:50.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:49 smithi064 ceph-mon[17163]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:50.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:49 smithi094 ceph-mon[20404]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:52.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:51 smithi064 ceph-mon[17163]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:52.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:51 smithi094 ceph-mon[20404]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:52.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:51 smithi087 ceph-mon[21100]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:53.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:52 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:53.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:52 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:53.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:52 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:53.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:53.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:53.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:53.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:52 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:53.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:52 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:53.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:52 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:54.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:54 smithi064 ceph-mon[17163]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:54.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:54 smithi094 ceph-mon[20404]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:54.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:54 smithi087 ceph-mon[21100]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:55.577 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 3 on host 'smithi087' 2024-09-02T11:57:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:55.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:55.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:55.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:55 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:55.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:55.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:55 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:57:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:57:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:56.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:57:56.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:56.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:55 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:57:56.318 DEBUG:teuthology.orchestra.run.smithi087:osd.3> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3.service 2024-09-02T11:57:56.320 INFO:tasks.cephadm:Deploying osd.4 on smithi087 with /dev/vg_nvme/lv_2... 2024-09-02T11:57:56.321 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-02T11:57:57.116 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:57:57.498 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 11:57:57 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3[30514]: 2024-09-02T11:57:57.310+0000 7f3a9e551640 -1 osd.3 0 log_to_monitors true 2024-09-02T11:57:57.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:57 smithi087 ceph-mon[21100]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:57.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:57 smithi087 ceph-mon[21100]: from='osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T11:57:57.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:57 smithi087 ceph-mon[21100]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T11:57:57.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:57 smithi064 ceph-mon[17163]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:57.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:57 smithi064 ceph-mon[17163]: from='osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T11:57:57.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:57 smithi064 ceph-mon[17163]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T11:57:57.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:57 smithi094 ceph-mon[20404]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:57.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:57 smithi094 ceph-mon[20404]: from='osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T11:57:57.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:57 smithi094 ceph-mon[20404]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T11:57:58.609 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-mon[21100]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-02T11:57:58.609 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-mon[21100]: osdmap e23: 4 total, 3 up, 4 in 2024-09-02T11:57:58.609 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-mon[21100]: from='osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:58.610 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:58.610 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-mon[21100]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:58.610 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3[30514]: 2024-09-02T11:57:58.601+0000 7f3a9a4d4640 -1 osd.3 0 waiting for initial osdmap 2024-09-02T11:57:58.610 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 11:57:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3[30514]: 2024-09-02T11:57:58.607+0000 7f3a962fb640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:57:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:58 smithi064 ceph-mon[17163]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-02T11:57:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:58 smithi064 ceph-mon[17163]: osdmap e23: 4 total, 3 up, 4 in 2024-09-02T11:57:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:58 smithi064 ceph-mon[17163]: from='osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:59.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:58 smithi064 ceph-mon[17163]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:59.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:58 smithi094 ceph-mon[20404]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-02T11:57:59.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:58 smithi094 ceph-mon[20404]: osdmap e23: 4 total, 3 up, 4 in 2024-09-02T11:57:59.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:58 smithi094 ceph-mon[20404]: from='osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:59.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:59.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:58 smithi094 ceph-mon[20404]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:57:59.787 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:59 smithi087 ceph-mon[21100]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:57:59.787 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:59 smithi087 ceph-mon[21100]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:57:59.787 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:59 smithi087 ceph-mon[21100]: osdmap e24: 4 total, 3 up, 4 in 2024-09-02T11:57:59.787 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:59 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:57:59.787 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:57:59 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:59 smithi064 ceph-mon[17163]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:58:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:59 smithi064 ceph-mon[17163]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:58:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:59 smithi064 ceph-mon[17163]: osdmap e24: 4 total, 3 up, 4 in 2024-09-02T11:58:00.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:59 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:00.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:57:59 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:59 smithi094 ceph-mon[20404]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T11:58:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:59 smithi094 ceph-mon[20404]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:58:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:59 smithi094 ceph-mon[20404]: osdmap e24: 4 total, 3 up, 4 in 2024-09-02T11:58:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:59 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:00.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:57:59 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:00.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581] boot 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: osdmap e25: 4 total, 4 up, 4 in 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi087 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi087 to 10592M 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:00.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:00.752 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:00 smithi087 ceph-mon[21100]: osdmap e26: 4 total, 4 up, 4 in 2024-09-02T11:58:01.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581] boot 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: osdmap e25: 4 total, 4 up, 4 in 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi087 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi087 to 10592M 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:01.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:01.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.096 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:00 smithi064 ceph-mon[17163]: osdmap e26: 4 total, 4 up, 4 in 2024-09-02T11:58:01.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:58:01.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:58:01.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: osd.3 [v2:172.21.15.87:6808/899665581,v1:172.21.15.87:6809/899665581] boot 2024-09-02T11:58:01.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: osdmap e25: 4 total, 4 up, 4 in 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi087 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi087 to 10592M 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:00 smithi094 ceph-mon[20404]: osdmap e26: 4 total, 4 up, 4 in 2024-09-02T11:58:01.139 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T11:58:01.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi087:vg_nvme/lv_2 2024-09-02T11:58:01.408 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:58:02.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:01 smithi094 ceph-mon[20404]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:02.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:01 smithi094 ceph-mon[20404]: osdmap e27: 4 total, 4 up, 4 in 2024-09-02T11:58:02.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:01 smithi087 ceph-mon[21100]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:02.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:01 smithi087 ceph-mon[21100]: osdmap e27: 4 total, 4 up, 4 in 2024-09-02T11:58:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:01 smithi064 ceph-mon[17163]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:01 smithi064 ceph-mon[17163]: osdmap e27: 4 total, 4 up, 4 in 2024-09-02T11:58:03.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:02 smithi087 ceph-mon[21100]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:03.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:02 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:03.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:02 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:03.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:02 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:02 smithi064 ceph-mon[17163]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:03.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:02 smithi094 ceph-mon[20404]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:03.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:03.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:03.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:04.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:03 smithi087 ceph-mon[21100]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:04.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:03 smithi064 ceph-mon[17163]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:04.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:03 smithi094 ceph-mon[20404]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:04 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/3643910832' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"} : dispatch 2024-09-02T11:58:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:04 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"} : dispatch 2024-09-02T11:58:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:04 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"}]': finished 2024-09-02T11:58:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:04 smithi087 ceph-mon[21100]: osdmap e28: 5 total, 4 up, 5 in 2024-09-02T11:58:05.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:05.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:04 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/3643910832' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"} : dispatch 2024-09-02T11:58:05.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:04 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"} : dispatch 2024-09-02T11:58:05.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:04 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"}]': finished 2024-09-02T11:58:05.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:04 smithi064 ceph-mon[17163]: osdmap e28: 5 total, 4 up, 5 in 2024-09-02T11:58:05.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:05.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:04 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/3643910832' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"} : dispatch 2024-09-02T11:58:05.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:04 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"} : dispatch 2024-09-02T11:58:05.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:04 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58217be9-55d9-43cd-9856-9c957bcc1107"}]': finished 2024-09-02T11:58:05.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:04 smithi094 ceph-mon[20404]: osdmap e28: 5 total, 4 up, 5 in 2024-09-02T11:58:05.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:06.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:05 smithi087 ceph-mon[21100]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:06.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:05 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/3106137501' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:06.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:05 smithi064 ceph-mon[17163]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:06.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:05 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/3106137501' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:06.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:05 smithi094 ceph-mon[20404]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:06.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:05 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/3106137501' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:08.157 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:07 smithi087 ceph-mon[21100]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-02T11:58:08.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:07 smithi064 ceph-mon[17163]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-02T11:58:08.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:07 smithi094 ceph-mon[20404]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-02T11:58:10.146 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:09 smithi064 ceph-mon[17163]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-02T11:58:10.166 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:09 smithi087 ceph-mon[21100]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-02T11:58:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:09 smithi094 ceph-mon[20404]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-02T11:58:11.916 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:11 smithi087 ceph-mon[21100]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T11:58:12.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:11 smithi064 ceph-mon[17163]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T11:58:12.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:11 smithi094 ceph-mon[20404]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T11:58:13.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-02T11:58:13.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:13.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:12 smithi087 ceph-mon[21100]: Deploying daemon osd.4 on smithi087 2024-09-02T11:58:13.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-02T11:58:13.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:13.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:12 smithi064 ceph-mon[17163]: Deploying daemon osd.4 on smithi087 2024-09-02T11:58:13.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-02T11:58:13.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:13.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:12 smithi094 ceph-mon[20404]: Deploying daemon osd.4 on smithi087 2024-09-02T11:58:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:13 smithi087 ceph-mon[21100]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-02T11:58:14.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:13 smithi064 ceph-mon[17163]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-02T11:58:14.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:13 smithi094 ceph-mon[20404]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-02T11:58:16.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:15 smithi087 ceph-mon[21100]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-02T11:58:16.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:15 smithi064 ceph-mon[17163]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-02T11:58:16.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:15 smithi094 ceph-mon[20404]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-02T11:58:18.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:17 smithi087 ceph-mon[21100]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-02T11:58:18.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:18.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:18.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:17 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:18.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:17 smithi064 ceph-mon[17163]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-02T11:58:18.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:18.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:18.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:17 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:17 smithi094 ceph-mon[20404]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-02T11:58:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:17 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:20.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:19 smithi087 ceph-mon[21100]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:20.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:19 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:20.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:19 smithi064 ceph-mon[17163]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:20.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:19 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:20.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:19 smithi094 ceph-mon[20404]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:20.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:19 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:20.932 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:20 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:20.932 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:20 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:20.933 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:20 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:20.933 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:20 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:20.971 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 4 on host 'smithi087' 2024-09-02T11:58:21.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:21.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:21.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:21.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:20 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:21.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:20 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:21.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:20 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:21.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:20 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:21.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:20 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:21.578 DEBUG:teuthology.orchestra.run.smithi087:osd.4> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4.service 2024-09-02T11:58:21.581 INFO:tasks.cephadm:Deploying osd.5 on smithi094 with /dev/vg_nvme/lv_4... 2024-09-02T11:58:21.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-02T11:58:21.831 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:58:22.063 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:21 smithi087 ceph-mon[21100]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:22.063 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:21 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:22.063 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:21 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:22.063 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:21 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:22.222 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:21 smithi094 ceph-mon[20404]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:22.222 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:21 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:22.222 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:21 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:22.223 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:21 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:22.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:21 smithi064 ceph-mon[17163]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:22.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:21 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:22.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:21 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:22.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:21 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:22.792 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 11:58:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T11:58:22.530+0000 7fa9541f7640 -1 osd.4 0 log_to_monitors true 2024-09-02T11:58:23.071 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:22 smithi087 ceph-mon[21100]: from='osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T11:58:23.071 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:22 smithi087 ceph-mon[21100]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T11:58:23.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:22 smithi094 ceph-mon[20404]: from='osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T11:58:23.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:22 smithi094 ceph-mon[20404]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T11:58:23.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:22 smithi064 ceph-mon[17163]: from='osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T11:58:23.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:22 smithi064 ceph-mon[17163]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T11:58:24.121 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:58:24.123 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:23 smithi087 ceph-mon[21100]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:24.123 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:23 smithi087 ceph-mon[21100]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-02T11:58:24.123 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:23 smithi087 ceph-mon[21100]: osdmap e29: 5 total, 4 up, 5 in 2024-09-02T11:58:24.123 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:24.123 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:23 smithi087 ceph-mon[21100]: from='osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:58:24.123 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:23 smithi087 ceph-mon[21100]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:58:24.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_4 2024-09-02T11:58:24.156 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:23 smithi094 ceph-mon[20404]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:24.156 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:23 smithi094 ceph-mon[20404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-02T11:58:24.156 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:23 smithi094 ceph-mon[20404]: osdmap e29: 5 total, 4 up, 5 in 2024-09-02T11:58:24.156 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:24.156 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:23 smithi094 ceph-mon[20404]: from='osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:58:24.156 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:23 smithi094 ceph-mon[20404]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:58:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:23 smithi064 ceph-mon[17163]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:23 smithi064 ceph-mon[17163]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-02T11:58:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:23 smithi064 ceph-mon[17163]: osdmap e29: 5 total, 4 up, 5 in 2024-09-02T11:58:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:23 smithi064 ceph-mon[17163]: from='osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:58:24.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:23 smithi064 ceph-mon[17163]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]} : dispatch 2024-09-02T11:58:24.387 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:58:24.449 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T11:58:24.326+0000 7fa95017a640 -1 osd.4 0 waiting for initial osdmap 2024-09-02T11:58:24.449 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T11:58:24.329+0000 7fa947f99640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:58:25.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:58:25.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: osdmap e30: 5 total, 4 up, 5 in 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='osd.4 ' entity='osd.4' 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:25.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:58:25.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: osdmap e30: 5 total, 4 up, 5 in 2024-09-02T11:58:25.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:25.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='osd.4 ' entity='osd.4' 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: osdmap e30: 5 total, 4 up, 5 in 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='osd.4 ' entity='osd.4' 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:25.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:25.346 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:25.346 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:26.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi087 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi087 to 7061M 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212] boot 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: osdmap e31: 5 total, 5 up, 5 in 2024-09-02T11:58:26.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:58:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:58:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi087 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi087 to 7061M 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212] boot 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: osdmap e31: 5 total, 5 up, 5 in 2024-09-02T11:58:26.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:26.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:58:26.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi087 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi087 to 7061M 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: osd.4 [v2:172.21.15.87:6816/2526415212,v1:172.21.15.87:6817/2526415212] boot 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: osdmap e31: 5 total, 5 up, 5 in 2024-09-02T11:58:26.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T11:58:27.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: osdmap e32: 5 total, 5 up, 5 in 2024-09-02T11:58:27.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:27.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/3719515449' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"} : dispatch 2024-09-02T11:58:27.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"} : dispatch 2024-09-02T11:58:27.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"}]': finished 2024-09-02T11:58:27.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: osdmap e33: 6 total, 5 up, 6 in 2024-09-02T11:58:27.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:27 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:27.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: osdmap e32: 5 total, 5 up, 5 in 2024-09-02T11:58:27.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:27.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/3719515449' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"} : dispatch 2024-09-02T11:58:27.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"} : dispatch 2024-09-02T11:58:27.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"}]': finished 2024-09-02T11:58:27.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: osdmap e33: 6 total, 5 up, 6 in 2024-09-02T11:58:27.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:27 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: osdmap e32: 5 total, 5 up, 5 in 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/3719515449' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"} : dispatch 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"} : dispatch 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc3a1f8a-0c14-492e-898a-b7d36a8c6f53"}]': finished 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: osdmap e33: 6 total, 5 up, 6 in 2024-09-02T11:58:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:27 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:28.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:28 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/706714219' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:28.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:28 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/706714219' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:28.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:28 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/706714219' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:29.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:29 smithi094 ceph-mon[20404]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:29.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:29 smithi087 ceph-mon[21100]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:29.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:29 smithi064 ceph-mon[17163]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:31.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:31 smithi064 ceph-mon[17163]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:31.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:31 smithi094 ceph-mon[20404]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:32.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:31 smithi087 ceph-mon[21100]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:33.820 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:33 smithi094 ceph-mon[20404]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:33.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:33 smithi064 ceph-mon[17163]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:34.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:33 smithi087 ceph-mon[21100]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:34.841 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-02T11:58:34.841 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:34 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:34.841 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:34 smithi094 ceph-mon[20404]: Deploying daemon osd.5 on smithi094 2024-09-02T11:58:34.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-02T11:58:34.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:34 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:34.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:34 smithi064 ceph-mon[17163]: Deploying daemon osd.5 on smithi094 2024-09-02T11:58:35.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-02T11:58:35.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:34 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:35.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:34 smithi087 ceph-mon[21100]: Deploying daemon osd.5 on smithi094 2024-09-02T11:58:35.742 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:35 smithi094 ceph-mon[20404]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:35.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:35 smithi064 ceph-mon[17163]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:36.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:35 smithi087 ceph-mon[21100]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:37.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:37 smithi064 ceph-mon[17163]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:37.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:37 smithi094 ceph-mon[20404]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:38.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:37 smithi087 ceph-mon[21100]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:39.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:39 smithi064 ceph-mon[17163]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:39.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:39.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:39.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:39 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:39.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:39 smithi094 ceph-mon[20404]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:39.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:39.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:39.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:39 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:40.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:39 smithi087 ceph-mon[21100]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:40.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:40.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:40.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:39 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:42 smithi087 ceph-mon[21100]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:42.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:42 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.547 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:42 smithi064 ceph-mon[17163]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:42.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:42.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:42.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:42 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:42 smithi094 ceph-mon[20404]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:42.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:42.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:42 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:43.312 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 5 on host 'smithi094' 2024-09-02T11:58:43.902 DEBUG:teuthology.orchestra.run.smithi094:osd.5> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5.service 2024-09-02T11:58:43.945 INFO:tasks.cephadm:Deploying osd.6 on smithi094 with /dev/vg_nvme/lv_3... 2024-09-02T11:58:43.945 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-02T11:58:44.147 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T11:58:44.113+0000 7f7499a03640 -1 osd.5 0 log_to_monitors true 2024-09-02T11:58:44.472 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-mon[20404]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:44.472 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:44.472 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:44.472 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:44.472 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-mon[20404]: from='osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T11:58:44.472 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:44 smithi094 ceph-mon[20404]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T11:58:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:44 smithi087 ceph-mon[21100]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:44 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:44 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:44 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:44 smithi087 ceph-mon[21100]: from='osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T11:58:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:44 smithi087 ceph-mon[21100]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T11:58:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:44 smithi064 ceph-mon[17163]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:58:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:44 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:44 smithi064 ceph-mon[17163]: from='osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T11:58:44.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:44 smithi064 ceph-mon[17163]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T11:58:45.062 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:58:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:45 smithi064 ceph-mon[17163]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-02T11:58:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:45 smithi064 ceph-mon[17163]: osdmap e34: 6 total, 5 up, 6 in 2024-09-02T11:58:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:45 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:45 smithi064 ceph-mon[17163]: from='osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:58:45.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:45 smithi064 ceph-mon[17163]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:58:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-mon[20404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-02T11:58:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-mon[20404]: osdmap e34: 6 total, 5 up, 6 in 2024-09-02T11:58:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-mon[20404]: from='osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:58:45.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-mon[20404]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:58:45.596 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T11:58:45.557+0000 7f7495986640 -1 osd.5 0 waiting for initial osdmap 2024-09-02T11:58:45.596 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 11:58:45 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T11:58:45.560+0000 7f748d7a5640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:58:45.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:45 smithi087 ceph-mon[21100]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-02T11:58:45.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:45 smithi087 ceph-mon[21100]: osdmap e34: 6 total, 5 up, 6 in 2024-09-02T11:58:45.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:45 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:45.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:45 smithi087 ceph-mon[21100]: from='osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:58:45.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:45 smithi087 ceph-mon[21100]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:58:46.579 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:46 smithi094 ceph-mon[20404]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:46.580 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:46 smithi094 ceph-mon[20404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:58:46.580 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:46 smithi094 ceph-mon[20404]: osdmap e35: 6 total, 5 up, 6 in 2024-09-02T11:58:46.580 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:46 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:46.580 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:46 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:46.580 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:46 smithi094 ceph-mon[20404]: from='osd.5 ' entity='osd.5' 2024-09-02T11:58:46.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:46 smithi064 ceph-mon[17163]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:46.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:46 smithi064 ceph-mon[17163]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:58:46.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:46 smithi064 ceph-mon[17163]: osdmap e35: 6 total, 5 up, 6 in 2024-09-02T11:58:46.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:46 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:46.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:46 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:46.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:46 smithi064 ceph-mon[17163]: from='osd.5 ' entity='osd.5' 2024-09-02T11:58:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:46 smithi087 ceph-mon[21100]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:46 smithi087 ceph-mon[21100]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:58:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:46 smithi087 ceph-mon[21100]: osdmap e35: 6 total, 5 up, 6 in 2024-09-02T11:58:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:46 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:46 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:46 smithi087 ceph-mon[21100]: from='osd.5 ' entity='osd.5' 2024-09-02T11:58:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:47 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:58:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:47 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:58:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:47 smithi064 ceph-mon[17163]: osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428] boot 2024-09-02T11:58:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:47 smithi064 ceph-mon[17163]: osdmap e36: 6 total, 6 up, 6 in 2024-09-02T11:58:47.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:47 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:58:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:47 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:58:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:47 smithi094 ceph-mon[20404]: osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428] boot 2024-09-02T11:58:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:47 smithi094 ceph-mon[20404]: osdmap e36: 6 total, 6 up, 6 in 2024-09-02T11:58:47.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:47.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:47 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:58:47.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:47 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:58:47.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:47.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:47 smithi087 ceph-mon[21100]: osd.5 [v2:172.21.15.94:6800/1385000428,v1:172.21.15.94:6801/1385000428] boot 2024-09-02T11:58:47.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:47 smithi087 ceph-mon[21100]: osdmap e36: 6 total, 6 up, 6 in 2024-09-02T11:58:47.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T11:58:48.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:48.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi094 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: osdmap e37: 6 total, 6 up, 6 in 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi094 to 21184M 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:48.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:48 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi094 2024-09-02T11:58:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: osdmap e37: 6 total, 6 up, 6 in 2024-09-02T11:58:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:48.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi094 to 21184M 2024-09-02T11:58:48.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:48.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:48.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:48 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T11:58:48.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi094 2024-09-02T11:58:48.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: osdmap e37: 6 total, 6 up, 6 in 2024-09-02T11:58:48.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi094 to 21184M 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:58:48.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:48 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:58:48.763 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:58:48.790 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_3 2024-09-02T11:58:49.031 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:58:49.447 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:49 smithi094 ceph-mon[20404]: osdmap e38: 6 total, 6 up, 6 in 2024-09-02T11:58:49.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:49 smithi064 ceph-mon[17163]: osdmap e38: 6 total, 6 up, 6 in 2024-09-02T11:58:49.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:49 smithi087 ceph-mon[21100]: osdmap e38: 6 total, 6 up, 6 in 2024-09-02T11:58:50.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:50 smithi064 ceph-mon[17163]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:50 smithi064 ceph-mon[17163]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:50 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:50 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:50 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:50 smithi094 ceph-mon[20404]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:50 smithi094 ceph-mon[20404]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:50 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:50 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:50.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:50 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:50 smithi087 ceph-mon[21100]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:58:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:50 smithi087 ceph-mon[21100]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:58:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:50 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:58:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:50 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:58:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:50 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:51.501 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:51 smithi094 ceph-mon[20404]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-02T11:58:51.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:51 smithi087 ceph-mon[21100]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-02T11:58:51.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:51 smithi064 ceph-mon[17163]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-09-02T11:58:52.652 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:52 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/33002065' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"} : dispatch 2024-09-02T11:58:52.652 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:52 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"} : dispatch 2024-09-02T11:58:52.652 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:52 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"}]': finished 2024-09-02T11:58:52.652 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:52 smithi087 ceph-mon[21100]: osdmap e39: 7 total, 6 up, 7 in 2024-09-02T11:58:52.652 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:52 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:58:52.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:52 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/33002065' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"} : dispatch 2024-09-02T11:58:52.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:52 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"} : dispatch 2024-09-02T11:58:52.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:52 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"}]': finished 2024-09-02T11:58:52.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:52 smithi064 ceph-mon[17163]: osdmap e39: 7 total, 6 up, 7 in 2024-09-02T11:58:52.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:52 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:58:52.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:52 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/33002065' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"} : dispatch 2024-09-02T11:58:52.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:52 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"} : dispatch 2024-09-02T11:58:52.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:52 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b436c92-04a1-4683-85fd-e7e4a1e406c3"}]': finished 2024-09-02T11:58:52.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:52 smithi094 ceph-mon[20404]: osdmap e39: 7 total, 6 up, 7 in 2024-09-02T11:58:52.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:52 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:58:53.616 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:53 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/1311921114' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:53.616 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:53 smithi094 ceph-mon[20404]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-02T11:58:53.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:53 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/1311921114' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:53.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:53 smithi087 ceph-mon[21100]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-02T11:58:53.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:53 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/1311921114' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:58:53.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:53 smithi064 ceph-mon[17163]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-02T11:58:55.708 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:55 smithi094 ceph-mon[20404]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-02T11:58:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:55 smithi064 ceph-mon[17163]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-02T11:58:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:55 smithi087 ceph-mon[21100]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-02T11:58:57.706 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:57 smithi094 ceph-mon[20404]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-02T11:58:57.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:57 smithi064 ceph-mon[17163]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-02T11:58:58.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:57 smithi087 ceph-mon[21100]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-02T11:58:59.669 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:59 smithi094 ceph-mon[20404]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-02T11:58:59.669 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:59 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-02T11:58:59.669 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:59 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:59.669 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:58:59 smithi094 ceph-mon[20404]: Deploying daemon osd.6 on smithi094 2024-09-02T11:58:59.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:59 smithi064 ceph-mon[17163]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-02T11:58:59.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:59 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-02T11:58:59.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:59 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:58:59.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:58:59 smithi064 ceph-mon[17163]: Deploying daemon osd.6 on smithi094 2024-09-02T11:59:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:59 smithi087 ceph-mon[21100]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-02T11:59:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:59 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-02T11:59:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:59 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:58:59 smithi087 ceph-mon[21100]: Deploying daemon osd.6 on smithi094 2024-09-02T11:59:01.705 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:01 smithi094 ceph-mon[20404]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:01.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:01 smithi064 ceph-mon[17163]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:02.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:01 smithi087 ceph-mon[21100]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:03.625 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:03 smithi094 ceph-mon[20404]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:03.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:03 smithi087 ceph-mon[21100]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:03.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:03 smithi064 ceph-mon[17163]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:04.816 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:04.816 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:04.816 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:04 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:04 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:05.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:05.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:05.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:04 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:06.477 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:06 smithi094 ceph-mon[20404]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:06.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:06 smithi087 ceph-mon[21100]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:06 smithi064 ceph-mon[17163]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:07.815 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:07 smithi094 ceph-mon[20404]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:07.815 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:07 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:07.815 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:07 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:07.815 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:07 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:07.815 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:07 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:07.815 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:07 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:07 smithi064 ceph-mon[17163]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:07 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:07 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:07 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:07 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:07.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:07 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:07 smithi087 ceph-mon[21100]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:07 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:07 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:07 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:07 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:07 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.088 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 6 on host 'smithi094' 2024-09-02T11:59:08.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:08.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.671 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.698 DEBUG:teuthology.orchestra.run.smithi094:osd.6> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6.service 2024-09-02T11:59:08.701 INFO:tasks.cephadm:Deploying osd.7 on smithi094 with /dev/vg_nvme/lv_2... 2024-09-02T11:59:08.701 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-02T11:59:08.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:08.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:08.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:09.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:09.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:09.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:09.172 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 11:59:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T11:59:08.945+0000 7fd75c26e640 -1 osd.6 0 log_to_monitors true 2024-09-02T11:59:09.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:09 smithi064 ceph-mon[17163]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:09.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:09 smithi064 ceph-mon[17163]: from='osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T11:59:09.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:09 smithi064 ceph-mon[17163]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T11:59:09.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:09 smithi094 ceph-mon[20404]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:09.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:09 smithi094 ceph-mon[20404]: from='osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T11:59:09.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:09 smithi094 ceph-mon[20404]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T11:59:09.917 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:59:10.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:09 smithi087 ceph-mon[21100]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:10.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:09 smithi087 ceph-mon[21100]: from='osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T11:59:10.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:09 smithi087 ceph-mon[21100]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T11:59:10.801 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-mon[20404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-02T11:59:10.801 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-mon[20404]: osdmap e40: 7 total, 6 up, 7 in 2024-09-02T11:59:10.801 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:10.801 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-mon[20404]: from='osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:10.801 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-mon[20404]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:10.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:10 smithi064 ceph-mon[17163]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-02T11:59:10.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:10 smithi064 ceph-mon[17163]: osdmap e40: 7 total, 6 up, 7 in 2024-09-02T11:59:10.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:10 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:10.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:10 smithi064 ceph-mon[17163]: from='osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:10.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:10 smithi064 ceph-mon[17163]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:11.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:10 smithi087 ceph-mon[21100]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-02T11:59:11.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:10 smithi087 ceph-mon[21100]: osdmap e40: 7 total, 6 up, 7 in 2024-09-02T11:59:11.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:10 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:11.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:10 smithi087 ceph-mon[21100]: from='osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:11.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:10 smithi087 ceph-mon[21100]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:11.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T11:59:10.800+0000 7fd7581f1640 -1 osd.6 0 waiting for initial osdmap 2024-09-02T11:59:11.097 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 11:59:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T11:59:10.805+0000 7fd750010640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:59:11.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:11 smithi064 ceph-mon[17163]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:11.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:11 smithi064 ceph-mon[17163]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:59:11.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:11 smithi064 ceph-mon[17163]: osdmap e41: 7 total, 6 up, 7 in 2024-09-02T11:59:11.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:11.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:11.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:11 smithi064 ceph-mon[17163]: from='osd.6 ' entity='osd.6' 2024-09-02T11:59:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:11 smithi094 ceph-mon[20404]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:11 smithi094 ceph-mon[20404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:59:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:11 smithi094 ceph-mon[20404]: osdmap e41: 7 total, 6 up, 7 in 2024-09-02T11:59:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:11 smithi094 ceph-mon[20404]: from='osd.6 ' entity='osd.6' 2024-09-02T11:59:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:11 smithi087 ceph-mon[21100]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:11 smithi087 ceph-mon[21100]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:59:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:11 smithi087 ceph-mon[21100]: osdmap e41: 7 total, 6 up, 7 in 2024-09-02T11:59:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:11 smithi087 ceph-mon[21100]: from='osd.6 ' entity='osd.6' 2024-09-02T11:59:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:59:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647] boot 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: osdmap e42: 7 total, 7 up, 7 in 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi094 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi094 to 10592M 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647] boot 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: osdmap e42: 7 total, 7 up, 7 in 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi094 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi094 to 10592M 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:13.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:59:13.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:59:13.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: osd.6 [v2:172.21.15.94:6808/3480856647,v1:172.21.15.94:6809/3480856647] boot 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: osdmap e42: 7 total, 7 up, 7 in 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi094 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:13.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:13.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:13.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi094 to 10592M 2024-09-02T11:59:13.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:13.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:13.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:13.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:13.476 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T11:59:13.504 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi094:vg_nvme/lv_2 2024-09-02T11:59:13.745 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:59:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:13 smithi064 ceph-mon[17163]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:13 smithi064 ceph-mon[17163]: osdmap e43: 7 total, 7 up, 7 in 2024-09-02T11:59:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:13 smithi094 ceph-mon[20404]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:13 smithi094 ceph-mon[20404]: osdmap e43: 7 total, 7 up, 7 in 2024-09-02T11:59:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:13 smithi087 ceph-mon[21100]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-09-02T11:59:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:13 smithi087 ceph-mon[21100]: osdmap e43: 7 total, 7 up, 7 in 2024-09-02T11:59:15.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:59:15.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:59:15.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:14 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:15.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:59:15.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:59:15.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:14 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:15.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T11:59:15.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T11:59:15.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:14 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:15.897 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:15 smithi094 ceph-mon[20404]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:59:15.898 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:15 smithi094 ceph-mon[20404]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:16.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:15 smithi064 ceph-mon[17163]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:59:16.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:15 smithi064 ceph-mon[17163]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:16.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:15 smithi087 ceph-mon[21100]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi094:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T11:59:16.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:15 smithi087 ceph-mon[21100]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:17.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:16 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/1338861538' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"} : dispatch 2024-09-02T11:59:17.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:16 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"} : dispatch 2024-09-02T11:59:17.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:16 smithi064 ceph-mon[17163]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"}]': finished 2024-09-02T11:59:17.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:16 smithi064 ceph-mon[17163]: osdmap e44: 8 total, 7 up, 8 in 2024-09-02T11:59:17.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:16 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:17.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:16 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/1338861538' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"} : dispatch 2024-09-02T11:59:17.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:16 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"} : dispatch 2024-09-02T11:59:17.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:16 smithi094 ceph-mon[20404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"}]': finished 2024-09-02T11:59:17.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:16 smithi094 ceph-mon[20404]: osdmap e44: 8 total, 7 up, 8 in 2024-09-02T11:59:17.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:16 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:17.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:16 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/1338861538' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"} : dispatch 2024-09-02T11:59:17.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:16 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"} : dispatch 2024-09-02T11:59:17.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:16 smithi087 ceph-mon[21100]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6602cfeb-51f2-4da4-b540-508ad469e4d5"}]': finished 2024-09-02T11:59:17.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:16 smithi087 ceph-mon[21100]: osdmap e44: 8 total, 7 up, 8 in 2024-09-02T11:59:17.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:16 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:18.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:17 smithi064 ceph-mon[17163]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:18.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:17 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/3537209546' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:59:18.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:17 smithi094 ceph-mon[20404]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:18.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:17 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/3537209546' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:59:18.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:17 smithi087 ceph-mon[21100]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:18.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:17 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/3537209546' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T11:59:20.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:19 smithi064 ceph-mon[17163]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:20.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:19 smithi094 ceph-mon[20404]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:20.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:19 smithi087 ceph-mon[21100]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:22.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:21 smithi094 ceph-mon[20404]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:22.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:21 smithi087 ceph-mon[21100]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:22.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:21 smithi064 ceph-mon[17163]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:24.087 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:23 smithi094 ceph-mon[20404]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:24.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:23 smithi087 ceph-mon[21100]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:23 smithi064 ceph-mon[17163]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:25.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-02T11:59:25.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:24 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:25.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:24 smithi094 ceph-mon[20404]: Deploying daemon osd.7 on smithi094 2024-09-02T11:59:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-02T11:59:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:24 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:25.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:24 smithi087 ceph-mon[21100]: Deploying daemon osd.7 on smithi094 2024-09-02T11:59:25.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-02T11:59:25.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:24 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:25.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:24 smithi064 ceph-mon[17163]: Deploying daemon osd.7 on smithi094 2024-09-02T11:59:25.949 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:25 smithi094 ceph-mon[20404]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:25 smithi087 ceph-mon[21100]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:26.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:25 smithi064 ceph-mon[17163]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:28.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:27 smithi094 ceph-mon[20404]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:28.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:27 smithi087 ceph-mon[21100]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:27 smithi064 ceph-mon[17163]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:29 smithi094 ceph-mon[20404]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:30.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:29 smithi087 ceph-mon[21100]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:30.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:30.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:30.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:30.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:29 smithi064 ceph-mon[17163]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:30.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:30.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:30.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:31 smithi094 ceph-mon[20404]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:32.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:32.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:32.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:31 smithi087 ceph-mon[21100]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:32.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:32.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:32.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:31 smithi064 ceph-mon[17163]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:32.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:32.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:32.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:32.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:33.040 INFO:teuthology.orchestra.run.smithi094.stdout:Created osd(s) 7 on host 'smithi094' 2024-09-02T11:59:33.664 DEBUG:teuthology.orchestra.run.smithi094:osd.7> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7.service 2024-09-02T11:59:33.666 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-02T11:59:33.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-09-02T11:59:33.904 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:33 smithi094 ceph-mon[20404]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:33.904 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:33 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:33.904 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:33 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:33.905 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:33 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:33.929 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:33 smithi064 ceph-mon[17163]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:33 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:34.241 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 11:59:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T11:59:34.101+0000 7f9e8862b640 -1 osd.7 0 log_to_monitors true 2024-09-02T11:59:34.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:33 smithi087 ceph-mon[21100]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:34.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:33 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T11:59:34.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:33 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:34.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:33 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:34.685 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:35.166 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:34 smithi094 ceph-mon[20404]: from='osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T11:59:35.166 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:34 smithi094 ceph-mon[20404]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T11:59:35.167 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:34 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2890865897' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T11:59:35.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:34 smithi064 ceph-mon[17163]: from='osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T11:59:35.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:34 smithi064 ceph-mon[17163]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T11:59:35.185 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:34 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2890865897' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T11:59:35.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:34 smithi087 ceph-mon[21100]: from='osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T11:59:35.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:34 smithi087 ceph-mon[21100]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T11:59:35.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:34 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2890865897' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T11:59:35.406 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":44,"num_osds":8,"num_up_osds":7,"osd_up_since":1725278351,"num_in_osds":8,"osd_in_since":1725278356,"num_remapped_pgs":0} 2024-09-02T11:59:36.096 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T11:59:35.879+0000 7f9e845ae640 -1 osd.7 0 waiting for initial osdmap 2024-09-02T11:59:36.097 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T11:59:35.884+0000 7f9e803d5640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T11:59:36.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-mon[20404]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:36.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-mon[20404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-02T11:59:36.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-mon[20404]: from='osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:36.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-mon[20404]: osdmap e45: 8 total, 7 up, 8 in 2024-09-02T11:59:36.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:36.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:35 smithi094 ceph-mon[20404]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:35 smithi087 ceph-mon[21100]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:35 smithi087 ceph-mon[21100]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-02T11:59:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:35 smithi087 ceph-mon[21100]: from='osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:35 smithi087 ceph-mon[21100]: osdmap e45: 8 total, 7 up, 8 in 2024-09-02T11:59:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:35 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:35 smithi087 ceph-mon[21100]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:35 smithi064 ceph-mon[17163]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:35 smithi064 ceph-mon[17163]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-02T11:59:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:35 smithi064 ceph-mon[17163]: from='osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:35 smithi064 ceph-mon[17163]: osdmap e45: 8 total, 7 up, 8 in 2024-09-02T11:59:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:35 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:36.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:35 smithi064 ceph-mon[17163]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]} : dispatch 2024-09-02T11:59:36.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-09-02T11:59:36.665 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:37.081 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: osdmap e46: 8 total, 7 up, 8 in 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:37.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:37.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: osdmap e46: 8 total, 7 up, 8 in 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:37.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.266 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:37.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi094", "root=default"]}]': finished 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: osdmap e46: 8 total, 7 up, 8 in 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T11:59:37.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T11:59:37.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T11:59:37.929 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":47,"num_osds":8,"num_up_osds":8,"osd_up_since":1725278376,"num_in_osds":8,"osd_in_since":1725278356,"num_remapped_pgs":0} 2024-09-02T11:59:37.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T11:59:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: purged_snaps scrub starts 2024-09-02T11:59:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: purged_snaps scrub ok 2024-09-02T11:59:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: Detected new or changed devices on smithi094 2024-09-02T11:59:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi094 to 7061M 2024-09-02T11:59:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014] boot 2024-09-02T11:59:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: osdmap e47: 8 total, 8 up, 8 in 2024-09-02T11:59:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:38.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:37 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/281250730' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T11:59:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: purged_snaps scrub starts 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: purged_snaps scrub ok 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: Detected new or changed devices on smithi094 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi094 to 7061M 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014] boot 2024-09-02T11:59:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: osdmap e47: 8 total, 8 up, 8 in 2024-09-02T11:59:38.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:38.251 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:37 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/281250730' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T11:59:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: purged_snaps scrub starts 2024-09-02T11:59:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: purged_snaps scrub ok 2024-09-02T11:59:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: Detected new or changed devices on smithi094 2024-09-02T11:59:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi094 to 7061M 2024-09-02T11:59:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-02T11:59:38.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:38.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: osd.7 [v2:172.21.15.94:6816/4279451014,v1:172.21.15.94:6817/4279451014] boot 2024-09-02T11:59:38.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: osdmap e47: 8 total, 8 up, 8 in 2024-09-02T11:59:38.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T11:59:38.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:37 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/281250730' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T11:59:38.821 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:38.822 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":48,"fsid":"e8e22764-6921-11ef-bcd4-c7b262605968","created":"2024-09-02T11:54:18.038584+0000","modified":"2024-09-02T11:59:37.888927+0000","last_up_change":"2024-09-02T11:59:36.877702+0000","last_in_change":"2024-09-02T11:59:16.595123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T11:57:36.919909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25b94b5e-a25e-49ac-ab44-e73f48429152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6803","nonce":2163384183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6805","nonce":2163384183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6809","nonce":2163384183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6807","nonce":2163384183}]},"public_addr":"172.21.15.64:6803/2163384183","cluster_addr":"172.21.15.64:6805/2163384183","heartbeat_back_addr":"172.21.15.64:6809/2163384183","heartbeat_front_addr":"172.21.15.64:6807/2163384183","state":["exists","up"]},{"osd":1,"uuid":"dd005fa8-f0f3-497a-a20c-b1706bf3c3de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6811","nonce":1982412301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6813","nonce":1982412301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6817","nonce":1982412301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6815","nonce":1982412301}]},"public_addr":"172.21.15.64:6811/1982412301","cluster_addr":"172.21.15.64:6813/1982412301","heartbeat_back_addr":"172.21.15.64:6817/1982412301","heartbeat_front_addr":"172.21.15.64:6815/1982412301","state":["exists","up"]},{"osd":2,"uuid":"f9a8efd8-9fae-4319-aa01-801cba9780a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6801","nonce":3018291796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6803","nonce":3018291796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6807","nonce":3018291796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6805","nonce":3018291796}]},"public_addr":"172.21.15.87:6801/3018291796","cluster_addr":"172.21.15.87:6803/3018291796","heartbeat_back_addr":"172.21.15.87:6807/3018291796","heartbeat_front_addr":"172.21.15.87:6805/3018291796","state":["exists","up"]},{"osd":3,"uuid":"59c9bbd9-a906-46a3-a870-40e7fad00805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6809","nonce":899665581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6811","nonce":899665581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6815","nonce":899665581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6813","nonce":899665581}]},"public_addr":"172.21.15.87:6809/899665581","cluster_addr":"172.21.15.87:6811/899665581","heartbeat_back_addr":"172.21.15.87:6815/899665581","heartbeat_front_addr":"172.21.15.87:6813/899665581","state":["exists","up"]},{"osd":4,"uuid":"58217be9-55d9-43cd-9856-9c957bcc1107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6817","nonce":2526415212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6819","nonce":2526415212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6823","nonce":2526415212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6821","nonce":2526415212}]},"public_addr":"172.21.15.87:6817/2526415212","cluster_addr":"172.21.15.87:6819/2526415212","heartbeat_back_addr":"172.21.15.87:6823/2526415212","heartbeat_front_addr":"172.21.15.87:6821/2526415212","state":["exists","up"]},{"osd":5,"uuid":"dc3a1f8a-0c14-492e-898a-b7d36a8c6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6801","nonce":1385000428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6803","nonce":1385000428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6807","nonce":1385000428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6805","nonce":1385000428}]},"public_addr":"172.21.15.94:6801/1385000428","cluster_addr":"172.21.15.94:6803/1385000428","heartbeat_back_addr":"172.21.15.94:6807/1385000428","heartbeat_front_addr":"172.21.15.94:6805/1385000428","state":["exists","up"]},{"osd":6,"uuid":"7b436c92-04a1-4683-85fd-e7e4a1e406c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6809","nonce":3480856647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6811","nonce":3480856647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6815","nonce":3480856647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6813","nonce":3480856647}]},"public_addr":"172.21.15.94:6809/3480856647","cluster_addr":"172.21.15.94:6811/3480856647","heartbeat_back_addr":"172.21.15.94:6815/3480856647","heartbeat_front_addr":"172.21.15.94:6813/3480856647","state":["exists","up"]},{"osd":7,"uuid":"6602cfeb-51f2-4da4-b540-508ad469e4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6817","nonce":4279451014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6819","nonce":4279451014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6823","nonce":4279451014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6821","nonce":4279451014}]},"public_addr":"172.21.15.94:6817/4279451014","cluster_addr":"172.21.15.94:6819/4279451014","heartbeat_back_addr":"172.21.15.94:6823/4279451014","heartbeat_front_addr":"172.21.15.94:6821/4279451014","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:56:44.473664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:09.437404+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:32.749377+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:58.268791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:58:23.529966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:58:45.067922+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:59:09.918172+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:59:35.124742+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/4139954857":"2024-09-03T11:55:04.433026+0000","172.21.15.64:0/636160678":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6801/1976263301":"2024-09-03T11:55:04.433026+0000","172.21.15.64:0/1077966414":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6800/1976263301":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6801/4177075706":"2024-09-03T11:54:40.299326+0000","172.21.15.64:6800/4177075706":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/1441727743":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/406053309":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/1783943718":"2024-09-03T11:54:40.299326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T11:59:39.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:38 smithi064 ceph-mon[17163]: osdmap e48: 8 total, 8 up, 8 in 2024-09-02T11:59:39.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:38 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1077498216' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:39.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:38 smithi087 ceph-mon[21100]: osdmap e48: 8 total, 8 up, 8 in 2024-09-02T11:59:39.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:38 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1077498216' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:39.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:38 smithi094 ceph-mon[20404]: osdmap e48: 8 total, 8 up, 8 in 2024-09-02T11:59:39.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:38 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1077498216' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:39.504 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T11:57:36.919909+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-02T11:59:39.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-02T11:59:39.751 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:40.153 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:39 smithi064 ceph-mon[17163]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:40.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:39 smithi087 ceph-mon[21100]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:40.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:39 smithi094 ceph-mon[20404]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:40.430 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-09-02T11:59:41.015 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:40 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3461583114' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-02T11:59:41.042 INFO:tasks.cephadm:Setting up client nodes... 2024-09-02T11:59:41.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T11:59:41.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:40 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3461583114' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-02T11:59:41.294 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:59:41.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:40 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3461583114' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-02T11:59:41.976 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-09-02T11:59:41.976 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCtqNVmbRr+ORAAwP5LmEub0Q21Wqnwg9FKiQ== 2024-09-02T11:59:42.246 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:41 smithi064 ceph-mon[17163]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:42.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:41 smithi087 ceph-mon[21100]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:42.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:41 smithi094 ceph-mon[20404]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:42.648 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T11:59:42.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-02T11:59:42.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-02T11:59:42.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T11:59:42.948 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:43.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:42 smithi064 ceph-mon[17163]: from='client.? 172.21.15.38:0/4291519450' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:43.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:42 smithi064 ceph-mon[17163]: from='client.? 172.21.15.38:0/4291519450' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:43.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:42 smithi087 ceph-mon[21100]: from='client.? 172.21.15.38:0/4291519450' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:43.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:42 smithi087 ceph-mon[21100]: from='client.? 172.21.15.38:0/4291519450' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:43.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:42 smithi094 ceph-mon[20404]: from='client.? 172.21.15.38:0/4291519450' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:43.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:42 smithi094 ceph-mon[20404]: from='client.? 172.21.15.38:0/4291519450' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:43.709 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-09-02T11:59:43.709 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCvqNVmgVckKhAAKTjK9CeNHI4jY3LCIKraWg== 2024-09-02T11:59:44.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:43 smithi087 ceph-mon[21100]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:44.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:43 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1315984979' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:44.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:43 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1315984979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:44.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:43 smithi064 ceph-mon[17163]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:44.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:43 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1315984979' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:44.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:43 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1315984979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:44.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:43 smithi094 ceph-mon[20404]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:44.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:43 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1315984979' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:44.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:43 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1315984979' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:44.424 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T11:59:44.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-02T11:59:44.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-02T11:59:44.460 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T11:59:44.712 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T11:59:45.431 INFO:teuthology.orchestra.run.smithi087.stdout:[client.2] 2024-09-02T11:59:45.431 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQCxqNVmtelzGRAALGvSyuhTIPs5H0l5jyqD9Q== 2024-09-02T11:59:46.124 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:45 smithi087 ceph-mon[21100]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:46.124 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:45 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/3258319207' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:46.124 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:45 smithi087 ceph-mon[21100]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:46.124 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:45 smithi087 ceph-mon[21100]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:46.125 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T11:59:46.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-09-02T11:59:46.126 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-02T11:59:46.164 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.3 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T11:59:46.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:45 smithi064 ceph-mon[17163]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:46.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:45 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/3258319207' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:46.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:45 smithi064 ceph-mon[17163]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:46.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:45 smithi064 ceph-mon[17163]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:46.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:45 smithi094 ceph-mon[20404]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:46.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:45 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/3258319207' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:46.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:45 smithi094 ceph-mon[20404]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:46.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:45 smithi094 ceph-mon[20404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:46.419 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T11:59:47.236 INFO:teuthology.orchestra.run.smithi094.stdout:[client.3] 2024-09-02T11:59:47.236 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCzqNVmqfvDDRAA9TeZzXF5H+OQX0JGq1Mc+w== 2024-09-02T11:59:47.851 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T11:59:47.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.client.3.keyring 2024-09-02T11:59:47.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-02T11:59:47.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.4 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T11:59:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:47 smithi094 ceph-mon[20404]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:47 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/3433483931' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:47 smithi094 ceph-mon[20404]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:47 smithi094 ceph-mon[20404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:48.139 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T11:59:48.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:47 smithi087 ceph-mon[21100]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:48.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:47 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/3433483931' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:48.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:47 smithi087 ceph-mon[21100]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:48.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:47 smithi087 ceph-mon[21100]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:48.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:47 smithi064 ceph-mon[17163]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:48.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:47 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/3433483931' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:48.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:47 smithi064 ceph-mon[17163]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:48.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:47 smithi064 ceph-mon[17163]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.3", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:48.812 INFO:teuthology.orchestra.run.smithi099.stdout:[client.4] 2024-09-02T11:59:48.812 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQC0qNVmPBchMBAAyrwcRw0tTWyN8i1Zz2ErgQ== 2024-09-02T11:59:49.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:48 smithi087 ceph-mon[21100]: from='client.? 172.21.15.99:0/1790054507' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:49.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:48 smithi087 ceph-mon[21100]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:49.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:48 smithi087 ceph-mon[21100]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:49.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:48 smithi064 ceph-mon[17163]: from='client.? 172.21.15.99:0/1790054507' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:49.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:48 smithi064 ceph-mon[17163]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:49.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:48 smithi064 ceph-mon[17163]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:49.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:48 smithi094 ceph-mon[20404]: from='client.? 172.21.15.99:0/1790054507' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:49.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:48 smithi094 ceph-mon[20404]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T11:59:49.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:48 smithi094 ceph-mon[20404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.4", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T11:59:49.579 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T11:59:49.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.4.keyring 2024-09-02T11:59:49.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.client.4.keyring 2024-09-02T11:59:49.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph config log 1 --format=json 2024-09-02T11:59:49.916 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:50.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:49 smithi087 ceph-mon[21100]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:50.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:49 smithi064 ceph-mon[17163]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:50.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:49 smithi094 ceph-mon[20404]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:50.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:51.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:50 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2567525642' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-02T11:59:51.320 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:50 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2567525642' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-02T11:59:51.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:50 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2567525642' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-02T11:59:51.355 INFO:teuthology.orchestra.run.smithi064.stdout:[{"version":22,"timestamp":"2024-09-02T11:59:36.478932+0000","name":"","changes":[{"name":"osd/host:smithi094/osd_memory_target","previous_value":"11106567372","new_value":"7404378248"}]}] 2024-09-02T11:59:51.355 INFO:tasks.ceph_manager:config epoch is 22 2024-09-02T11:59:51.355 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-02T11:59:51.356 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-02T11:59:51.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph mgr dump --format=json 2024-09-02T11:59:51.356 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-02T11:59:51.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T11:59:51.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T11:59:51.470 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T11:59:51.511 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T11:59:51.588 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T11:59:51.612 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:51.664 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T11:59:51.705 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T11:59:51.784 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T11:59:51.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T11:59:51.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T11:59:51.927 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T11:59:51.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T11:59:52.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:51 smithi087 ceph-mon[21100]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:52.307 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:52.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:51 smithi064 ceph-mon[17163]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:52.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:51 smithi094 ceph-mon[20404]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:53.015 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":14,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":3012055107},{"type":"v1","addr":"172.21.15.64:6801","nonce":3012055107}]},"active_addr":"172.21.15.64:6801/3012055107","active_change":"2024-09-02T11:55:04.433134+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2776942821}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2341474435}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2212330072}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":98214064}]}]} 2024-09-02T11:59:53.018 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-02T11:59:53.018 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-02T11:59:53.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T11:59:53.262 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:53.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:53 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/681813641' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-02T11:59:53.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:53 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/681813641' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-02T11:59:53.461 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:53 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/681813641' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-02T11:59:53.854 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:53.854 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":48,"fsid":"e8e22764-6921-11ef-bcd4-c7b262605968","created":"2024-09-02T11:54:18.038584+0000","modified":"2024-09-02T11:59:37.888927+0000","last_up_change":"2024-09-02T11:59:36.877702+0000","last_in_change":"2024-09-02T11:59:16.595123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T11:57:36.919909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25b94b5e-a25e-49ac-ab44-e73f48429152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6803","nonce":2163384183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6805","nonce":2163384183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6809","nonce":2163384183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6807","nonce":2163384183}]},"public_addr":"172.21.15.64:6803/2163384183","cluster_addr":"172.21.15.64:6805/2163384183","heartbeat_back_addr":"172.21.15.64:6809/2163384183","heartbeat_front_addr":"172.21.15.64:6807/2163384183","state":["exists","up"]},{"osd":1,"uuid":"dd005fa8-f0f3-497a-a20c-b1706bf3c3de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6811","nonce":1982412301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6813","nonce":1982412301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6817","nonce":1982412301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6815","nonce":1982412301}]},"public_addr":"172.21.15.64:6811/1982412301","cluster_addr":"172.21.15.64:6813/1982412301","heartbeat_back_addr":"172.21.15.64:6817/1982412301","heartbeat_front_addr":"172.21.15.64:6815/1982412301","state":["exists","up"]},{"osd":2,"uuid":"f9a8efd8-9fae-4319-aa01-801cba9780a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6801","nonce":3018291796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6803","nonce":3018291796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6807","nonce":3018291796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6805","nonce":3018291796}]},"public_addr":"172.21.15.87:6801/3018291796","cluster_addr":"172.21.15.87:6803/3018291796","heartbeat_back_addr":"172.21.15.87:6807/3018291796","heartbeat_front_addr":"172.21.15.87:6805/3018291796","state":["exists","up"]},{"osd":3,"uuid":"59c9bbd9-a906-46a3-a870-40e7fad00805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6809","nonce":899665581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6811","nonce":899665581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6815","nonce":899665581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6813","nonce":899665581}]},"public_addr":"172.21.15.87:6809/899665581","cluster_addr":"172.21.15.87:6811/899665581","heartbeat_back_addr":"172.21.15.87:6815/899665581","heartbeat_front_addr":"172.21.15.87:6813/899665581","state":["exists","up"]},{"osd":4,"uuid":"58217be9-55d9-43cd-9856-9c957bcc1107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6817","nonce":2526415212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6819","nonce":2526415212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6823","nonce":2526415212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6821","nonce":2526415212}]},"public_addr":"172.21.15.87:6817/2526415212","cluster_addr":"172.21.15.87:6819/2526415212","heartbeat_back_addr":"172.21.15.87:6823/2526415212","heartbeat_front_addr":"172.21.15.87:6821/2526415212","state":["exists","up"]},{"osd":5,"uuid":"dc3a1f8a-0c14-492e-898a-b7d36a8c6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6801","nonce":1385000428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6803","nonce":1385000428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6807","nonce":1385000428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6805","nonce":1385000428}]},"public_addr":"172.21.15.94:6801/1385000428","cluster_addr":"172.21.15.94:6803/1385000428","heartbeat_back_addr":"172.21.15.94:6807/1385000428","heartbeat_front_addr":"172.21.15.94:6805/1385000428","state":["exists","up"]},{"osd":6,"uuid":"7b436c92-04a1-4683-85fd-e7e4a1e406c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6809","nonce":3480856647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6811","nonce":3480856647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6815","nonce":3480856647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6813","nonce":3480856647}]},"public_addr":"172.21.15.94:6809/3480856647","cluster_addr":"172.21.15.94:6811/3480856647","heartbeat_back_addr":"172.21.15.94:6815/3480856647","heartbeat_front_addr":"172.21.15.94:6813/3480856647","state":["exists","up"]},{"osd":7,"uuid":"6602cfeb-51f2-4da4-b540-508ad469e4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6817","nonce":4279451014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6819","nonce":4279451014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6823","nonce":4279451014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6821","nonce":4279451014}]},"public_addr":"172.21.15.94:6817/4279451014","cluster_addr":"172.21.15.94:6819/4279451014","heartbeat_back_addr":"172.21.15.94:6823/4279451014","heartbeat_front_addr":"172.21.15.94:6821/4279451014","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:56:44.473664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:09.437404+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:32.749377+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:58.268791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:58:23.529966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:58:45.067922+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:59:09.918172+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:59:35.124742+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/4139954857":"2024-09-03T11:55:04.433026+0000","172.21.15.64:0/636160678":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6801/1976263301":"2024-09-03T11:55:04.433026+0000","172.21.15.64:0/1077966414":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6800/1976263301":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6801/4177075706":"2024-09-03T11:54:40.299326+0000","172.21.15.64:6800/4177075706":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/1441727743":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/406053309":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/1783943718":"2024-09-03T11:54:40.299326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T11:59:54.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:54 smithi094 ceph-mon[20404]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:54.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:54 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/404512954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:54.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:54 smithi064 ceph-mon[17163]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:54.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:54 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/404512954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:54.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:54 smithi087 ceph-mon[21100]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:54.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:54 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/404512954' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:54.607 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-02T11:59:54.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T11:59:54.854 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:55.489 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T11:59:55.489 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":48,"fsid":"e8e22764-6921-11ef-bcd4-c7b262605968","created":"2024-09-02T11:54:18.038584+0000","modified":"2024-09-02T11:59:37.888927+0000","last_up_change":"2024-09-02T11:59:36.877702+0000","last_in_change":"2024-09-02T11:59:16.595123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T11:57:36.919909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"25b94b5e-a25e-49ac-ab44-e73f48429152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6803","nonce":2163384183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6805","nonce":2163384183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6809","nonce":2163384183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2163384183},{"type":"v1","addr":"172.21.15.64:6807","nonce":2163384183}]},"public_addr":"172.21.15.64:6803/2163384183","cluster_addr":"172.21.15.64:6805/2163384183","heartbeat_back_addr":"172.21.15.64:6809/2163384183","heartbeat_front_addr":"172.21.15.64:6807/2163384183","state":["exists","up"]},{"osd":1,"uuid":"dd005fa8-f0f3-497a-a20c-b1706bf3c3de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6811","nonce":1982412301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6813","nonce":1982412301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6817","nonce":1982412301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":1982412301},{"type":"v1","addr":"172.21.15.64:6815","nonce":1982412301}]},"public_addr":"172.21.15.64:6811/1982412301","cluster_addr":"172.21.15.64:6813/1982412301","heartbeat_back_addr":"172.21.15.64:6817/1982412301","heartbeat_front_addr":"172.21.15.64:6815/1982412301","state":["exists","up"]},{"osd":2,"uuid":"f9a8efd8-9fae-4319-aa01-801cba9780a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6801","nonce":3018291796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6803","nonce":3018291796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6807","nonce":3018291796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":3018291796},{"type":"v1","addr":"172.21.15.87:6805","nonce":3018291796}]},"public_addr":"172.21.15.87:6801/3018291796","cluster_addr":"172.21.15.87:6803/3018291796","heartbeat_back_addr":"172.21.15.87:6807/3018291796","heartbeat_front_addr":"172.21.15.87:6805/3018291796","state":["exists","up"]},{"osd":3,"uuid":"59c9bbd9-a906-46a3-a870-40e7fad00805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6809","nonce":899665581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6811","nonce":899665581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6815","nonce":899665581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":899665581},{"type":"v1","addr":"172.21.15.87:6813","nonce":899665581}]},"public_addr":"172.21.15.87:6809/899665581","cluster_addr":"172.21.15.87:6811/899665581","heartbeat_back_addr":"172.21.15.87:6815/899665581","heartbeat_front_addr":"172.21.15.87:6813/899665581","state":["exists","up"]},{"osd":4,"uuid":"58217be9-55d9-43cd-9856-9c957bcc1107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6817","nonce":2526415212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6819","nonce":2526415212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6823","nonce":2526415212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":2526415212},{"type":"v1","addr":"172.21.15.87:6821","nonce":2526415212}]},"public_addr":"172.21.15.87:6817/2526415212","cluster_addr":"172.21.15.87:6819/2526415212","heartbeat_back_addr":"172.21.15.87:6823/2526415212","heartbeat_front_addr":"172.21.15.87:6821/2526415212","state":["exists","up"]},{"osd":5,"uuid":"dc3a1f8a-0c14-492e-898a-b7d36a8c6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6801","nonce":1385000428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6803","nonce":1385000428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6807","nonce":1385000428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":1385000428},{"type":"v1","addr":"172.21.15.94:6805","nonce":1385000428}]},"public_addr":"172.21.15.94:6801/1385000428","cluster_addr":"172.21.15.94:6803/1385000428","heartbeat_back_addr":"172.21.15.94:6807/1385000428","heartbeat_front_addr":"172.21.15.94:6805/1385000428","state":["exists","up"]},{"osd":6,"uuid":"7b436c92-04a1-4683-85fd-e7e4a1e406c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6809","nonce":3480856647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6811","nonce":3480856647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6815","nonce":3480856647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":3480856647},{"type":"v1","addr":"172.21.15.94:6813","nonce":3480856647}]},"public_addr":"172.21.15.94:6809/3480856647","cluster_addr":"172.21.15.94:6811/3480856647","heartbeat_back_addr":"172.21.15.94:6815/3480856647","heartbeat_front_addr":"172.21.15.94:6813/3480856647","state":["exists","up"]},{"osd":7,"uuid":"6602cfeb-51f2-4da4-b540-508ad469e4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6817","nonce":4279451014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6819","nonce":4279451014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6823","nonce":4279451014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":4279451014},{"type":"v1","addr":"172.21.15.94:6821","nonce":4279451014}]},"public_addr":"172.21.15.94:6817/4279451014","cluster_addr":"172.21.15.94:6819/4279451014","heartbeat_back_addr":"172.21.15.94:6823/4279451014","heartbeat_front_addr":"172.21.15.94:6821/4279451014","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:56:44.473664+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:09.437404+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:32.749377+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:57:58.268791+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:58:23.529966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:58:45.067922+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:59:09.918172+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T11:59:35.124742+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/4139954857":"2024-09-03T11:55:04.433026+0000","172.21.15.64:0/636160678":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6801/1976263301":"2024-09-03T11:55:04.433026+0000","172.21.15.64:0/1077966414":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6800/1976263301":"2024-09-03T11:55:04.433026+0000","172.21.15.64:6801/4177075706":"2024-09-03T11:54:40.299326+0000","172.21.15.64:6800/4177075706":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/1441727743":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/406053309":"2024-09-03T11:54:40.299326+0000","172.21.15.64:0/1783943718":"2024-09-03T11:54:40.299326+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T11:59:56.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-02T11:59:56.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-02T11:59:56.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-02T11:59:56.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-02T11:59:56.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-02T11:59:56.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-02T11:59:56.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-02T11:59:56.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-02T11:59:56.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:56 smithi064 ceph-mon[17163]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:56.346 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:56 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3500306878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:56 smithi094 ceph-mon[20404]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:56 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3500306878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:56.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:56 smithi087 ceph-mon[21100]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:56.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:56 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3500306878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T11:59:56.544 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.784 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.784 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.785 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.785 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.786 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.786 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.789 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T11:59:56.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T11:59:57.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T11:59:57.066 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T11:59:57.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T11:59:57.182 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T11:59:57.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T11:59:57.298 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T11:59:57.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T11:59:57.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T11:59:57.506 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T11:59:57.540 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T11:59:57.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T11:59:58.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 11:59:58 smithi064 ceph-mon[17163]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:58.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 11:59:58 smithi094 ceph-mon[20404]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:58.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 11:59:58 smithi087 ceph-mon[21100]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T11:59:58.612 INFO:teuthology.orchestra.run.smithi064.stdout:133143986198 2024-09-02T11:59:58.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-02T12:00:00.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:00 smithi064 ceph-mon[17163]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:00.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:00 smithi064 ceph-mon[17163]: overall HEALTH_OK 2024-09-02T12:00:00.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:00 smithi094 ceph-mon[20404]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:00.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:00 smithi094 ceph-mon[20404]: overall HEALTH_OK 2024-09-02T12:00:00.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:00 smithi087 ceph-mon[21100]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:00.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:00 smithi087 ceph-mon[21100]: overall HEALTH_OK 2024-09-02T12:00:01.413 INFO:teuthology.orchestra.run.smithi064.stdout:107374182425 2024-09-02T12:00:01.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-02T12:00:01.629 INFO:teuthology.orchestra.run.smithi064.stdout:180388626445 2024-09-02T12:00:01.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-02T12:00:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:02 smithi064 ceph-mon[17163]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:02.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:02 smithi094 ceph-mon[20404]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:02 smithi087 ceph-mon[21100]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:02.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:02.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:02.712 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:02.761 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:02.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:02.869 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:02.888 INFO:teuthology.orchestra.run.smithi064.stdout:55834574884 2024-09-02T12:00:02.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-02T12:00:02.916 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:02.993 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:03.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:03.099 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:03.131 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:03.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:201863462920 2024-09-02T12:00:03.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-02T12:00:04.071 INFO:teuthology.orchestra.run.smithi064.stdout:34359738410 2024-09-02T12:00:04.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-02T12:00:04.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:04 smithi064 ceph-mon[17163]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:04.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:04 smithi094 ceph-mon[20404]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:04.477 INFO:teuthology.orchestra.run.smithi064.stdout:154618822673 2024-09-02T12:00:04.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-02T12:00:04.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:04 smithi087 ceph-mon[21100]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:05.143 INFO:teuthology.orchestra.run.smithi064.stdout:77309411361 2024-09-02T12:00:05.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-02T12:00:05.156 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:05.424 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:05.882 INFO:teuthology.orchestra.run.smithi064.stdout:133143986199 2024-09-02T12:00:06.326 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:06 smithi064 ceph-mon[17163]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:06.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:06 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2650207854' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T12:00:06.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:06 smithi094 ceph-mon[20404]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:06.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:06 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2650207854' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T12:00:06.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:06 smithi087 ceph-mon[21100]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:06.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:06 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2650207854' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T12:00:06.605 INFO:teuthology.orchestra.run.smithi064.stdout:107374182426 2024-09-02T12:00:07.120 INFO:teuthology.orchestra.run.smithi064.stdout:180388626446 2024-09-02T12:00:07.172 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:07 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1425740952' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T12:00:07.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:07 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1425740952' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T12:00:07.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:07 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1425740952' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T12:00:07.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626445 got 180388626446 for osd.6 2024-09-02T12:00:07.758 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:07.935 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:08.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:08.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:08 smithi064 ceph-mon[17163]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:08.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:08 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2147539603' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T12:00:08.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:08.339 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:08 smithi087 ceph-mon[21100]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:08.339 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:08 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2147539603' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T12:00:08.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:08 smithi094 ceph-mon[20404]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:08.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:08 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2147539603' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T12:00:08.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:08.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:08.498 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:08.536 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:08.583 INFO:teuthology.orchestra.run.smithi064.stdout:55834574885 2024-09-02T12:00:08.611 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:08.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:08.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:08.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:08.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:08.794 INFO:teuthology.orchestra.run.smithi064.stdout:201863462921 2024-09-02T12:00:09.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:09 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/644708980' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T12:00:09.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:09 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/496348181' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T12:00:09.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:09 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/644708980' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T12:00:09.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:09 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/496348181' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T12:00:09.370 INFO:teuthology.orchestra.run.smithi064.stdout:34359738412 2024-09-02T12:00:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:09 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/644708980' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T12:00:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:09 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/496348181' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T12:00:09.947 INFO:teuthology.orchestra.run.smithi064.stdout:77309411362 2024-09-02T12:00:10.218 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:10 smithi064 ceph-mon[17163]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:10.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:10 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3906685837' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T12:00:10.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:10 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1894160549' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T12:00:10.245 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462920 got 201863462921 for osd.7 2024-09-02T12:00:10.245 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:10 smithi094 ceph-mon[20404]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:10 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3906685837' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T12:00:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:10 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1894160549' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T12:00:10.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:10 smithi087 ceph-mon[21100]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:10.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:10 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3906685837' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T12:00:10.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:10 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1894160549' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T12:00:11.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738410 got 34359738412 for osd.0 2024-09-02T12:00:11.351 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:11.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574884 got 55834574885 for osd.1 2024-09-02T12:00:11.425 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:11.820 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986198 got 133143986199 for osd.4 2024-09-02T12:00:11.820 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:12.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182425 got 107374182426 for osd.3 2024-09-02T12:00:12.191 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:12.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:12 smithi064 ceph-mon[17163]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:12.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:12 smithi094 ceph-mon[20404]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:12 smithi087 ceph-mon[21100]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:12.616 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411361 got 77309411362 for osd.2 2024-09-02T12:00:12.616 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:12.634 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:13.280 INFO:teuthology.orchestra.run.smithi064.stdout:154618822676 2024-09-02T12:00:13.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:13.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:13.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:13.961 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:14.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:14.051 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822673 got 154618822676 for osd.5 2024-09-02T12:00:14.051 DEBUG:teuthology.parallel:result is None 2024-09-02T12:00:14.052 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-02T12:00:14.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T12:00:14.111 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:14.148 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:14.223 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:14 smithi087 ceph-mon[21100]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:14 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/1886141586' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T12:00:14.296 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:14 smithi094 ceph-mon[20404]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:14.296 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:14 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/1886141586' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T12:00:14.296 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:14.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:14.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:14 smithi064 ceph-mon[17163]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:14.319 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:14 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/1886141586' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T12:00:14.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:14.363 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:14.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T12:00:14.937 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-02T12:00:15.678 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-02T12:00:14.490006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":229896,"kb_used_data":5324,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749731320,"statfs":{"total":767960285184,"available":767724871680,"internally_reserved":0,"allocated":5451776,"data_stored":3257536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":229756466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002071"},"pg_stats":[{"pgid":"1.0","version":"22'92","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-02T11:59:37.901669+0000","last_change":"2024-09-02T11:58:48.345339+0000","last_active":"2024-09-02T11:59:37.901669+0000","last_peered":"2024-09-02T11:59:37.901669+0000","last_clean":"2024-09-02T11:59:37.901669+0000","last_became_active":"2024-09-02T11:58:48.339335+0000","last_became_peered":"2024-09-02T11:58:48.339335+0000","last_unstale":"2024-09-02T11:59:37.901669+0000","last_undegraded":"2024-09-02T11:59:37.901669+0000","last_fullsized":"2024-09-02T11:59:37.901669+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T11:57:37.740768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T11:57:37.740768+0000","last_clean_scrub_stamp":"2024-09-02T11:57:37.740768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-03T16:40:57.460779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717676,"statfs":{"total":95995035648,"available":95966900224,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31648,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713504,"statfs":{"total":95995035648,"available":95962628096,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822676,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":776172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182428,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":776172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411363,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1052672,"data_stored":776172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738413,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T12:00:15.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T12:00:15.926 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:16.295 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:16 smithi064 ceph-mon[17163]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:16.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:16 smithi064 ceph-mon[17163]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:00:16.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:16 smithi094 ceph-mon[20404]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:16.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:16 smithi094 ceph-mon[20404]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:00:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:16 smithi087 ceph-mon[21100]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:16 smithi087 ceph-mon[21100]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:00:16.560 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T12:00:16.560 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-02T12:00:17.225 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-02T12:00:16.490262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":229896,"kb_used_data":5324,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749731320,"statfs":{"total":767960285184,"available":767724871680,"internally_reserved":0,"allocated":5451776,"data_stored":3257536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":229756466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002044"},"pg_stats":[{"pgid":"1.0","version":"22'92","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-02T11:59:37.901669+0000","last_change":"2024-09-02T11:58:48.345339+0000","last_active":"2024-09-02T11:59:37.901669+0000","last_peered":"2024-09-02T11:59:37.901669+0000","last_clean":"2024-09-02T11:59:37.901669+0000","last_became_active":"2024-09-02T11:58:48.339335+0000","last_became_peered":"2024-09-02T11:58:48.339335+0000","last_unstale":"2024-09-02T11:59:37.901669+0000","last_undegraded":"2024-09-02T11:59:37.901669+0000","last_fullsized":"2024-09-02T11:59:37.901669+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T11:57:37.740768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T11:57:37.740768+0000","last_clean_scrub_stamp":"2024-09-02T11:57:37.740768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-03T16:40:57.460779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":47,"seq":201863462923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27476,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717676,"statfs":{"total":95995035648,"available":95966900224,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31648,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713504,"statfs":{"total":95995035648,"available":95962628096,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822676,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":776172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986202,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182429,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":776172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411364,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1052672,"data_stored":776172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738413,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":458752,"data_stored":185804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T12:00:17.226 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-02T12:00:17.226 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-02T12:00:17.226 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-02T12:00:17.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph health --format=json 2024-09-02T12:00:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:18.144 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T12:00:18.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-02T12:00:18.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:18 smithi064 ceph-mon[17163]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:18.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:18 smithi064 ceph-mon[17163]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:00:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:18 smithi094 ceph-mon[20404]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:18 smithi094 ceph-mon[20404]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:00:18.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:18 smithi087 ceph-mon[21100]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:18.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:18 smithi087 ceph-mon[21100]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:00:18.834 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-02T12:00:18.834 INFO:tasks.cephadm:Setup complete, yielding 2024-09-02T12:00:18.834 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-02T12:00:18.844 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi064.front.sepia.ceph.com 2024-09-02T12:00:18.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch status' 2024-09-02T12:00:19.091 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:19.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:19 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3204251034' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-02T12:00:19.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:19 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3204251034' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-02T12:00:19.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:19.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:19.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:19 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3204251034' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-02T12:00:19.532 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:19.569 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:19.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:19.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:19.750 INFO:teuthology.orchestra.run.smithi064.stdout:Backend: cephadm 2024-09-02T12:00:19.750 INFO:teuthology.orchestra.run.smithi064.stdout:Available: Yes 2024-09-02T12:00:19.750 INFO:teuthology.orchestra.run.smithi064.stdout:Paused: No 2024-09-02T12:00:19.757 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:19.832 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:19.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:19.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:19.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:20.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:20.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:20 smithi087 ceph-mon[21100]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:20.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:20 smithi064 ceph-mon[17163]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:20.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:20 smithi094 ceph-mon[20404]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:20.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-02T12:00:20.786 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:21.163 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:21 smithi064 ceph-mon[17163]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:21.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:21 smithi094 ceph-mon[20404]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:21.451 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-02T12:00:21.451 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi064 *:9283,8765 running (5m) 3m ago 5m 503M - 19.3.0-4585-gb59673c4 b92d9064031c 2020f5ad642a 2024-09-02T12:00:21.451 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (5m) 3m ago 6m 40.6M 2048M 19.3.0-4585-gb59673c4 b92d9064031c 538852d87dbb 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi087 running (4m) 2m ago 4m 36.4M 2048M 19.3.0-4585-gb59673c4 b92d9064031c e7448cffa948 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi094 running (4m) 50s ago 4m 39.9M 2048M 19.3.0-4585-gb59673c4 b92d9064031c 9b7c8f05dfb3 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (3m) 3m ago 3m 25.7M 8544M 19.3.0-4585-gb59673c4 b92d9064031c 5ffc06ce4035 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (3m) 3m ago 3m 11.7M 8544M 19.3.0-4585-gb59673c4 b92d9064031c c2fc1b33e96f 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi087 running (2m) 2m ago 2m 43.1M 7061M 19.3.0-4585-gb59673c4 b92d9064031c 277c5fcec278 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi087 running (2m) 2m ago 2m 33.4M 7061M 19.3.0-4585-gb59673c4 b92d9064031c d2e243a77dd3 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi087 running (2m) 2m ago 2m 11.3M 7061M 19.3.0-4585-gb59673c4 b92d9064031c a4c97f18837f 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi094 running (102s) 50s ago 102s 39.4M 7061M 19.3.0-4585-gb59673c4 b92d9064031c 53daeb17158c 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi094 running (77s) 50s ago 77s 39.6M 7061M 19.3.0-4585-gb59673c4 b92d9064031c a21034e7f87c 2024-09-02T12:00:21.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi094 running (52s) 50s ago 52s 12.1M 7061M 19.3.0-4585-gb59673c4 b92d9064031c 49beed78224d 2024-09-02T12:00:21.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:21 smithi087 ceph-mon[21100]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:22.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-02T12:00:22.246 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:22 smithi064 ceph-mon[17163]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:22.315 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:22.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:22 smithi094 ceph-mon[20404]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:22.401 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:22 smithi087 ceph-mon[21100]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:22.953 INFO:teuthology.orchestra.run.smithi064.stdout:HOST ADDR LABELS STATUS 2024-09-02T12:00:22.953 INFO:teuthology.orchestra.run.smithi064.stdout:smithi038 172.21.15.38 2024-09-02T12:00:22.953 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 172.21.15.64 2024-09-02T12:00:22.953 INFO:teuthology.orchestra.run.smithi064.stdout:smithi087 172.21.15.87 2024-09-02T12:00:22.953 INFO:teuthology.orchestra.run.smithi064.stdout:smithi094 172.21.15.94 2024-09-02T12:00:22.954 INFO:teuthology.orchestra.run.smithi064.stdout:smithi099 172.21.15.99 2024-09-02T12:00:22.954 INFO:teuthology.orchestra.run.smithi064.stdout:5 hosts in cluster 2024-09-02T12:00:23.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:23 smithi064 ceph-mon[17163]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:23.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:23 smithi094 ceph-mon[20404]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:23.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:23 smithi087 ceph-mon[21100]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:23.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-02T12:00:23.845 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:24.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:24 smithi064 ceph-mon[17163]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:24.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:24 smithi064 ceph-mon[17163]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:24.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:24 smithi094 ceph-mon[20404]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:24.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:24 smithi094 ceph-mon[20404]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:24.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:24 smithi087 ceph-mon[21100]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:24.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:24 smithi087 ceph-mon[21100]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:24.501 INFO:teuthology.orchestra.run.smithi064.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-02T12:00:24.501 INFO:teuthology.orchestra.run.smithi064.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 4m ago Has a FileSystem, LVM detected 2024-09-02T12:00:24.501 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-02T12:00:24.501 INFO:teuthology.orchestra.run.smithi064.stdout:smithi087 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NQ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-02T12:00:24.501 INFO:teuthology.orchestra.run.smithi064.stdout:smithi094 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513001L480DGN 447G No 48s ago Has a FileSystem, LVM detected 2024-09-02T12:00:24.501 INFO:teuthology.orchestra.run.smithi064.stdout:smithi099 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040101400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-09-02T12:00:25.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:25.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:25.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:25.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:25.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- bash -c 'ceph osd lspools' 2024-09-02T12:00:25.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:25.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:25.373 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:25.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:25.487 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:25.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:25.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:25.593 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:25.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:26.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:26 smithi064 ceph-mon[17163]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:26.173 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:26 smithi064 ceph-mon[17163]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:26.173 INFO:teuthology.orchestra.run.smithi064.stdout:1 .mgr 2024-09-02T12:00:26.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:26 smithi094 ceph-mon[20404]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:26.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:26 smithi094 ceph-mon[20404]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:26.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:26 smithi087 ceph-mon[21100]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:26.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:26 smithi087 ceph-mon[21100]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:26.874 INFO:teuthology.run_tasks:Running task nvmeof... 2024-09-02T12:00:26.889 INFO:tasks.nvmeof:[nvmeof]: deploying nvmeof gateway... 2024-09-02T12:00:26.889 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.a on smithi064 2024-09-02T12:00:26.889 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.b on smithi087 2024-09-02T12:00:26.889 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.c on smithi094 2024-09-02T12:00:26.889 INFO:tasks.nvmeof:Adding ceph.nvmeof.nvmeof.d on smithi099 2024-09-02T12:00:26.890 INFO:tasks.nvmeof:[nvmeof]: ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/nvmeof:1.2 2024-09-02T12:00:26.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/nvmeof:1.2 2024-09-02T12:00:27.137 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:27.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:27 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2579165210' entity='client.admin' cmd={"prefix": "osd lspools"} : dispatch 2024-09-02T12:00:27.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:27 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2579165210' entity='client.admin' cmd={"prefix": "osd lspools"} : dispatch 2024-09-02T12:00:27.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:27 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2579165210' entity='client.admin' cmd={"prefix": "osd lspools"} : dispatch 2024-09-02T12:00:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:28 smithi064 ceph-mon[17163]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:28 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/249776240' entity='client.admin' 2024-09-02T12:00:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:00:28.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:28 smithi094 ceph-mon[20404]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:28.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:28 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/249776240' entity='client.admin' 2024-09-02T12:00:28.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:00:28.388 INFO:tasks.nvmeof:[nvmeof]: ceph osd pool create mypool 2024-09-02T12:00:28.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph osd pool create mypool 2024-09-02T12:00:28.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:28 smithi087 ceph-mon[21100]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:28.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:28 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/249776240' entity='client.admin' 2024-09-02T12:00:28.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:00:28.632 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:29.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:29.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:00:29.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:00:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:29.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:29.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:00:29.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:30.221 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'mypool' created 2024-09-02T12:00:30.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:30 smithi087 ceph-mon[21100]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:30.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:30 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3149561960' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "mypool"} : dispatch 2024-09-02T12:00:30.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:30 smithi064 ceph-mon[17163]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:30.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:30 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3149561960' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "mypool"} : dispatch 2024-09-02T12:00:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:30 smithi094 ceph-mon[20404]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:30 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3149561960' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "mypool"} : dispatch 2024-09-02T12:00:30.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:30.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:30.773 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:30.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:30.889 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:30.924 INFO:tasks.nvmeof:[nvmeof]: rbd pool init mypool 2024-09-02T12:00:30.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd pool init mypool 2024-09-02T12:00:30.963 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:31.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:31.079 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:31.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:31.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:31.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:31.250 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:31 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3149561960' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2024-09-02T12:00:31.250 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:31 smithi094 ceph-mon[20404]: osdmap e49: 8 total, 8 up, 8 in 2024-09-02T12:00:31.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:31.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:31 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3149561960' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2024-09-02T12:00:31.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:31 smithi064 ceph-mon[17163]: osdmap e49: 8 total, 8 up, 8 in 2024-09-02T12:00:31.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:31 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3149561960' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2024-09-02T12:00:31.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:31 smithi087 ceph-mon[21100]: osdmap e49: 8 total, 8 up, 8 in 2024-09-02T12:00:32.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:32 smithi087 ceph-mon[21100]: pgmap v202: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:32.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:32 smithi087 ceph-mon[21100]: osdmap e50: 8 total, 8 up, 8 in 2024-09-02T12:00:32.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:32 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2535508379' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "mypool","app": "rbd"} : dispatch 2024-09-02T12:00:32.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:32 smithi064 ceph-mon[17163]: pgmap v202: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:32.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:32 smithi064 ceph-mon[17163]: osdmap e50: 8 total, 8 up, 8 in 2024-09-02T12:00:32.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:32 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2535508379' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "mypool","app": "rbd"} : dispatch 2024-09-02T12:00:32.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:32 smithi094 ceph-mon[20404]: pgmap v202: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:32.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:32 smithi094 ceph-mon[20404]: osdmap e50: 8 total, 8 up, 8 in 2024-09-02T12:00:32.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:32 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2535508379' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "mypool","app": "rbd"} : dispatch 2024-09-02T12:00:33.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:33 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2535508379' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "mypool","app": "rbd"}]': finished 2024-09-02T12:00:33.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:33 smithi087 ceph-mon[21100]: osdmap e51: 8 total, 8 up, 8 in 2024-09-02T12:00:33.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:33 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2535508379' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "mypool","app": "rbd"}]': finished 2024-09-02T12:00:33.526 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:33 smithi094 ceph-mon[20404]: osdmap e51: 8 total, 8 up, 8 in 2024-09-02T12:00:33.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:33 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2535508379' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "mypool","app": "rbd"}]': finished 2024-09-02T12:00:33.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:33 smithi064 ceph-mon[17163]: osdmap e51: 8 total, 8 up, 8 in 2024-09-02T12:00:34.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:34 smithi064 ceph-mon[17163]: pgmap v205: 33 pgs: 6 creating+peering, 15 unknown, 12 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:34.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:34 smithi064 ceph-mon[17163]: osdmap e52: 8 total, 8 up, 8 in 2024-09-02T12:00:34.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:34 smithi094 ceph-mon[20404]: pgmap v205: 33 pgs: 6 creating+peering, 15 unknown, 12 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:34.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:34 smithi094 ceph-mon[20404]: osdmap e52: 8 total, 8 up, 8 in 2024-09-02T12:00:34.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:34 smithi087 ceph-mon[21100]: pgmap v205: 33 pgs: 6 creating+peering, 15 unknown, 12 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:34.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:34 smithi087 ceph-mon[21100]: osdmap e52: 8 total, 8 up, 8 in 2024-09-02T12:00:35.020 INFO:tasks.nvmeof:[nvmeof]: ceph orch apply nvmeof mypool mygroup0 2024-09-02T12:00:35.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch apply nvmeof mypool mygroup0 --placement 'smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d' 2024-09-02T12:00:35.275 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:35.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:35 smithi064 ceph-mon[17163]: osdmap e53: 8 total, 8 up, 8 in 2024-09-02T12:00:35.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:35 smithi094 ceph-mon[20404]: osdmap e53: 8 total, 8 up, 8 in 2024-09-02T12:00:35.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:35 smithi087 ceph-mon[21100]: osdmap e53: 8 total, 8 up, 8 in 2024-09-02T12:00:35.995 INFO:teuthology.orchestra.run.smithi064.stdout:Scheduled nvmeof.mypool.mygroup0 update... 2024-09-02T12:00:36.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:36.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:36.403 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:36.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:36.512 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:36 smithi087 ceph-mon[21100]: pgmap v208: 33 pgs: 6 creating+peering, 5 unknown, 22 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:36.512 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:36.512 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:36 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:00:36.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:36.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:36 smithi064 ceph-mon[17163]: pgmap v208: 33 pgs: 6 creating+peering, 5 unknown, 22 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:36 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:00:36.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:36 smithi094 ceph-mon[20404]: pgmap v208: 33 pgs: 6 creating+peering, 5 unknown, 22 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:00:36.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:36.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:36 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:00:36.624 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:36.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:36.717 INFO:tasks.nvmeof:[nvmeof]: creating 60 images 2024-09-02T12:00:36.718 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage1 --size 8192 2024-09-02T12:00:36.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage1 --size 8192 2024-09-02T12:00:36.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:36.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:36.974 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:36.984 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:37.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "group": "mygroup0", "placement": "smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: Saving service nvmeof.mypool.mygroup0 spec with placement smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:37.311 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:37 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:37.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "group": "mygroup0", "placement": "smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:37.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: Saving service nvmeof.mypool.mygroup0 spec with placement smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d 2024-09-02T12:00:37.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:37.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:00:37.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:37.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:37.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:37.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:37 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:37.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "group": "mygroup0", "placement": "smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:00:37.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: Saving service nvmeof.mypool.mygroup0 spec with placement smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d 2024-09-02T12:00:37.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:37.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:00:37.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:37.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:37.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.b", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:37.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:37 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:38.070 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage2 --size 8192 2024-09-02T12:00:38.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage2 --size 8192 2024-09-02T12:00:38.319 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:38.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:38 smithi064 ceph-mon[17163]: Deploying daemon nvmeof.nvmeof.b on smithi087 2024-09-02T12:00:38.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:38 smithi064 ceph-mon[17163]: pgmap v209: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 389 B/s wr, 0 op/s 2024-09-02T12:00:38.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:38 smithi094 ceph-mon[20404]: Deploying daemon nvmeof.nvmeof.b on smithi087 2024-09-02T12:00:38.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:38 smithi094 ceph-mon[20404]: pgmap v209: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 389 B/s wr, 0 op/s 2024-09-02T12:00:38.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:38 smithi087 ceph-mon[21100]: Deploying daemon nvmeof.nvmeof.b on smithi087 2024-09-02T12:00:38.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:38 smithi087 ceph-mon[21100]: pgmap v209: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 389 B/s wr, 0 op/s 2024-09-02T12:00:39.569 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage3 --size 8192 2024-09-02T12:00:39.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage3 --size 8192 2024-09-02T12:00:39.816 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:40.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:40 smithi064 ceph-mon[17163]: pgmap v210: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 490 B/s rd, 817 B/s wr, 1 op/s 2024-09-02T12:00:40.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:40 smithi094 ceph-mon[20404]: pgmap v210: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 490 B/s rd, 817 B/s wr, 1 op/s 2024-09-02T12:00:40.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:40 smithi087 ceph-mon[21100]: pgmap v210: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 490 B/s rd, 817 B/s wr, 1 op/s 2024-09-02T12:00:41.082 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage4 --size 8192 2024-09-02T12:00:41.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage4 --size 8192 2024-09-02T12:00:41.327 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:42.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:42.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:42.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:42.193 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:42.266 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:42.342 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:42.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:42 smithi064 ceph-mon[17163]: pgmap v211: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 895 B/s wr, 2 op/s 2024-09-02T12:00:42.380 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:42.456 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:42.463 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage5 --size 8192 2024-09-02T12:00:42.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage5 --size 8192 2024-09-02T12:00:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:42 smithi087 ceph-mon[21100]: pgmap v211: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 895 B/s wr, 2 op/s 2024-09-02T12:00:42.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:42.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:42 smithi094 ceph-mon[20404]: pgmap v211: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 895 B/s wr, 2 op/s 2024-09-02T12:00:42.599 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:42.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:42.711 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:43.746 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage6 --size 8192 2024-09-02T12:00:43.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage6 --size 8192 2024-09-02T12:00:43.991 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:44.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:43 smithi064 ceph-mon[17163]: pgmap v212: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 885 B/s wr, 2 op/s 2024-09-02T12:00:44.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:43 smithi064 ceph-mon[17163]: osdmap e54: 8 total, 8 up, 8 in 2024-09-02T12:00:44.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:43 smithi087 ceph-mon[21100]: pgmap v212: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 885 B/s wr, 2 op/s 2024-09-02T12:00:44.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:43 smithi087 ceph-mon[21100]: osdmap e54: 8 total, 8 up, 8 in 2024-09-02T12:00:44.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:43 smithi094 ceph-mon[20404]: pgmap v212: 33 pgs: 33 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 885 B/s wr, 2 op/s 2024-09-02T12:00:44.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:43 smithi094 ceph-mon[20404]: osdmap e54: 8 total, 8 up, 8 in 2024-09-02T12:00:44.997 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage7 --size 8192 2024-09-02T12:00:44.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage7 --size 8192 2024-09-02T12:00:45.242 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:46.189 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:45 smithi087 ceph-mon[21100]: pgmap v214: 33 pgs: 33 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-02T12:00:46.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:45 smithi064 ceph-mon[17163]: pgmap v214: 33 pgs: 33 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-02T12:00:46.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:45 smithi094 ceph-mon[20404]: pgmap v214: 33 pgs: 33 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-02T12:00:46.442 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage8 --size 8192 2024-09-02T12:00:46.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage8 --size 8192 2024-09-02T12:00:46.697 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:47.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:47.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:47.774 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:47.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:47.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:47.932 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: pgmap v215: 33 pgs: 33 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:47.933 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:47 smithi064 ceph-mon[17163]: Deploying daemon nvmeof.nvmeof.d on smithi099 2024-09-02T12:00:47.934 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage9 --size 8192 2024-09-02T12:00:47.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage9 --size 8192 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: pgmap v215: 33 pgs: 33 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:47.941 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:47 smithi087 ceph-mon[21100]: Deploying daemon nvmeof.nvmeof.d on smithi099 2024-09-02T12:00:47.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: pgmap v215: 33 pgs: 33 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.d", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:47.977 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:47 smithi094 ceph-mon[20404]: Deploying daemon nvmeof.nvmeof.d on smithi099 2024-09-02T12:00:47.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:48.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:48.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:48.177 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:48.180 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:48.208 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:48.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:49.268 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage10 --size 8192 2024-09-02T12:00:49.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage10 --size 8192 2024-09-02T12:00:49.514 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:49.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:49 smithi064 ceph-mon[17163]: pgmap v216: 33 pgs: 33 active+clean; 580 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:00:50.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:49 smithi087 ceph-mon[21100]: pgmap v216: 33 pgs: 33 active+clean; 580 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:00:50.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:49 smithi094 ceph-mon[20404]: pgmap v216: 33 pgs: 33 active+clean; 580 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:00:50.717 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage11 --size 8192 2024-09-02T12:00:50.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage11 --size 8192 2024-09-02T12:00:50.962 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:52.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:51 smithi087 ceph-mon[21100]: pgmap v217: 33 pgs: 33 active+clean; 581 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-09-02T12:00:52.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:51 smithi064 ceph-mon[17163]: pgmap v217: 33 pgs: 33 active+clean; 581 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-09-02T12:00:52.054 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage12 --size 8192 2024-09-02T12:00:52.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage12 --size 8192 2024-09-02T12:00:52.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:51 smithi094 ceph-mon[20404]: pgmap v217: 33 pgs: 33 active+clean; 581 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-09-02T12:00:52.299 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:53.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:53.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:53.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:53.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:53.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:53.521 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage13 --size 8192 2024-09-02T12:00:53.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage13 --size 8192 2024-09-02T12:00:53.566 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:53.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:53.679 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:53.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:53.766 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:53.775 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:53 smithi064 ceph-mon[17163]: pgmap v218: 33 pgs: 33 active+clean; 583 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.3 KiB/s wr, 9 op/s 2024-09-02T12:00:53.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:53.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:53.843 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:53 smithi094 ceph-mon[20404]: pgmap v218: 33 pgs: 33 active+clean; 583 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.3 KiB/s wr, 9 op/s 2024-09-02T12:00:53.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:00:54.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:53 smithi087 ceph-mon[21100]: pgmap v218: 33 pgs: 33 active+clean; 583 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.3 KiB/s wr, 9 op/s 2024-09-02T12:00:54.867 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage14 --size 8192 2024-09-02T12:00:54.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage14 --size 8192 2024-09-02T12:00:55.115 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:55 smithi064 ceph-mon[17163]: pgmap v219: 33 pgs: 33 active+clean; 583 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-09-02T12:00:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:55 smithi087 ceph-mon[21100]: pgmap v219: 33 pgs: 33 active+clean; 583 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-09-02T12:00:56.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:55 smithi094 ceph-mon[20404]: pgmap v219: 33 pgs: 33 active+clean; 583 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-09-02T12:00:56.180 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage15 --size 8192 2024-09-02T12:00:56.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage15 --size 8192 2024-09-02T12:00:56.425 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:57.576 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage16 --size 8192 2024-09-02T12:00:57.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage16 --size 8192 2024-09-02T12:00:57.819 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:58.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: pgmap v220: 33 pgs: 33 active+clean; 584 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-02T12:00:58.181 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.182 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.182 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.182 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:58.182 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:58.182 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:58.182 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:00:58 smithi064 ceph-mon[17163]: Deploying daemon nvmeof.nvmeof.c on smithi094 2024-09-02T12:00:58.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: pgmap v220: 33 pgs: 33 active+clean; 584 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:58.347 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:00:58 smithi094 ceph-mon[20404]: Deploying daemon nvmeof.nvmeof.c on smithi094 2024-09-02T12:00:58.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: pgmap v220: 33 pgs: 33 active+clean; 584 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-02T12:00:58.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:00:58.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:00:58.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.c", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:00:58.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:00:58.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:00:58 smithi087 ceph-mon[21100]: Deploying daemon nvmeof.nvmeof.c on smithi094 2024-09-02T12:00:58.793 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage17 --size 8192 2024-09-02T12:00:58.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage17 --size 8192 2024-09-02T12:00:58.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:00:58.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:00:58.993 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:00:59.030 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:00:59.042 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:00:59.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:00:59.180 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:00:59.226 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:00:59.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:00:59.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:00:59.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:00:59.455 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:00:59.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:00.031 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage18 --size 8192 2024-09-02T12:01:00.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage18 --size 8192 2024-09-02T12:01:00.277 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:00.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:00 smithi064 ceph-mon[17163]: pgmap v221: 33 pgs: 33 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-09-02T12:01:00.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:00 smithi087 ceph-mon[21100]: pgmap v221: 33 pgs: 33 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-09-02T12:01:00.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:00 smithi094 ceph-mon[20404]: pgmap v221: 33 pgs: 33 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-09-02T12:01:01.358 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage19 --size 8192 2024-09-02T12:01:01.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage19 --size 8192 2024-09-02T12:01:01.607 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:02 smithi064 ceph-mon[17163]: pgmap v222: 33 pgs: 33 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:02 smithi087 ceph-mon[21100]: pgmap v222: 33 pgs: 33 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:02.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:02 smithi094 ceph-mon[20404]: pgmap v222: 33 pgs: 33 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:02.723 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage20 --size 8192 2024-09-02T12:01:02.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage20 --size 8192 2024-09-02T12:01:02.970 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:04.015 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage21 --size 8192 2024-09-02T12:01:04.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage21 --size 8192 2024-09-02T12:01:04.272 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:04.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:04 smithi064 ceph-mon[17163]: pgmap v223: 33 pgs: 33 active+clean; 586 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-09-02T12:01:04.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:04 smithi087 ceph-mon[21100]: pgmap v223: 33 pgs: 33 active+clean; 586 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-09-02T12:01:04.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:04.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:04.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:04 smithi094 ceph-mon[20404]: pgmap v223: 33 pgs: 33 active+clean; 586 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-09-02T12:01:04.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:04.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:04.748 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:04.822 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:04.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:04.940 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:05.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:05.046 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:05.082 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:05.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:05.369 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage22 --size 8192 2024-09-02T12:01:05.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage22 --size 8192 2024-09-02T12:01:05.615 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:06.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:06 smithi064 ceph-mon[17163]: pgmap v224: 33 pgs: 33 active+clean; 586 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-09-02T12:01:06.410 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:06 smithi094 ceph-mon[20404]: pgmap v224: 33 pgs: 33 active+clean; 586 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-09-02T12:01:06.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:06 smithi087 ceph-mon[21100]: pgmap v224: 33 pgs: 33 active+clean; 586 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-09-02T12:01:06.752 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage23 --size 8192 2024-09-02T12:01:06.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage23 --size 8192 2024-09-02T12:01:07.008 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:08.359 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage24 --size 8192 2024-09-02T12:01:08.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage24 --size 8192 2024-09-02T12:01:08.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: pgmap v225: 33 pgs: 33 active+clean; 587 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:08.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:08 smithi064 ceph-mon[17163]: Deploying daemon nvmeof.nvmeof.a on smithi064 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: pgmap v225: 33 pgs: 33 active+clean; 587 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:08 smithi094 ceph-mon[20404]: Deploying daemon nvmeof.nvmeof.a on smithi064 2024-09-02T12:01:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: pgmap v225: 33 pgs: 33 active+clean; 587 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]} : dispatch 2024-09-02T12:01:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.nvmeof.a", "caps": ["mon", "profile rbd", "osd", "profile rbd"]}]': finished 2024-09-02T12:01:08.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:08.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:08 smithi087 ceph-mon[21100]: Deploying daemon nvmeof.nvmeof.a on smithi064 2024-09-02T12:01:09.785 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage25 --size 8192 2024-09-02T12:01:09.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage25 --size 8192 2024-09-02T12:01:10.042 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:10.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:10.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:10.275 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:10.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:10.387 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:10.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:10 smithi064 ceph-mon[17163]: pgmap v226: 33 pgs: 33 active+clean; 588 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-02T12:01:10.462 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:10.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:10.569 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:10 smithi094 ceph-mon[20404]: pgmap v226: 33 pgs: 33 active+clean; 588 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-02T12:01:10.580 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:10.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:10.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:10.723 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:10 smithi087 ceph-mon[21100]: pgmap v226: 33 pgs: 33 active+clean; 588 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-09-02T12:01:10.734 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:10.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:11.834 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage26 --size 8192 2024-09-02T12:01:11.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage26 --size 8192 2024-09-02T12:01:12.159 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:12.364 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:12 smithi064 ceph-mon[17163]: pgmap v227: 33 pgs: 33 active+clean; 589 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:12 smithi094 ceph-mon[20404]: pgmap v227: 33 pgs: 33 active+clean; 589 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:12 smithi087 ceph-mon[21100]: pgmap v227: 33 pgs: 33 active+clean; 589 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:14.566 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:14 smithi064 ceph-mon[17163]: pgmap v228: 33 pgs: 33 active+clean; 589 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-09-02T12:01:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:14 smithi094 ceph-mon[20404]: pgmap v228: 33 pgs: 33 active+clean; 589 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-09-02T12:01:14.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:14 smithi087 ceph-mon[21100]: pgmap v228: 33 pgs: 33 active+clean; 589 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-09-02T12:01:15.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:15.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:15.928 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:15.966 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:16.044 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:16.115 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:16.153 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:16.228 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:16.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:16.337 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:16.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:16.397 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:16 smithi094 ceph-mon[20404]: pgmap v229: 33 pgs: 33 active+clean; 590 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:16.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:16.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:16 smithi064 ceph-mon[17163]: pgmap v229: 33 pgs: 33 active+clean; 590 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:16.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:16 smithi087 ceph-mon[21100]: pgmap v229: 33 pgs: 33 active+clean; 590 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:01:17.269 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage27 --size 8192 2024-09-02T12:01:17.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage27 --size 8192 2024-09-02T12:01:18.119 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:18.462 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:18 smithi064 ceph-mon[17163]: pgmap v230: 33 pgs: 33 active+clean; 590 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-09-02T12:01:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:18 smithi094 ceph-mon[20404]: pgmap v230: 33 pgs: 33 active+clean; 590 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-09-02T12:01:18.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:18 smithi087 ceph-mon[21100]: pgmap v230: 33 pgs: 33 active+clean; 590 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-09-02T12:01:20.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:20 smithi064 ceph-mon[17163]: pgmap v231: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-09-02T12:01:20.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:20 smithi094 ceph-mon[20404]: pgmap v231: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-09-02T12:01:20.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:20 smithi087 ceph-mon[21100]: pgmap v231: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-09-02T12:01:21.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:21.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:21.556 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:21.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:21.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:21.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:21.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:21.860 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:21.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:21.969 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:22.000 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:22.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:22.302 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage28 --size 8192 2024-09-02T12:01:22.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage28 --size 8192 2024-09-02T12:01:22.571 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:22 smithi064 ceph-mon[17163]: pgmap v232: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-02T12:01:22.571 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:22 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.571 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:22 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.571 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:22 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.571 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:22 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.571 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:22 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:01:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:22 smithi094 ceph-mon[20404]: pgmap v232: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-02T12:01:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:22 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:22 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:22 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:22 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:22 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:01:22.605 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:22 smithi087 ceph-mon[21100]: pgmap v232: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-02T12:01:22.605 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:22 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.605 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:22 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.606 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:22 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.606 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:22 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:22.606 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:22 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:01:23.364 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2024-09-02T12:01:23.365 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.365 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.365 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.365 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.365 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.365 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:23 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.369 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:23.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2024-09-02T12:01:23.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:23 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: Checking pool "mypool" exists for service nvmeof.mypool.mygroup0 2024-09-02T12:01:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:23.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:23 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:24.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:24 smithi064 ceph-mon[17163]: pgmap v233: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-02T12:01:24.427 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:24 smithi064 ceph-mon[17163]: mgrmap e15: x(active, since 6m) 2024-09-02T12:01:24.455 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage29 --size 8192 2024-09-02T12:01:24.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage29 --size 8192 2024-09-02T12:01:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:24 smithi094 ceph-mon[20404]: pgmap v233: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-02T12:01:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:24 smithi094 ceph-mon[20404]: mgrmap e15: x(active, since 6m) 2024-09-02T12:01:24.715 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:24 smithi087 ceph-mon[21100]: pgmap v233: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-02T12:01:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:24 smithi087 ceph-mon[21100]: mgrmap e15: x(active, since 6m) 2024-09-02T12:01:25.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: pgmap v234: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-02T12:01:25.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: nvmeof daemon_check_post [(nvmeof.nvmeof.a), (nvmeof.nvmeof.b), (nvmeof.nvmeof.c), (nvmeof.nvmeof.d)] 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.a', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.b', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.783 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:25 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2436766328' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:25.808 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage30 --size 8192 2024-09-02T12:01:25.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage30 --size 8192 2024-09-02T12:01:25.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: pgmap v234: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-02T12:01:25.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: nvmeof daemon_check_post [(nvmeof.nvmeof.a), (nvmeof.nvmeof.b), (nvmeof.nvmeof.c), (nvmeof.nvmeof.d)] 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.a', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.b', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:25 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2436766328' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:25.850 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: pgmap v234: 33 pgs: 33 active+clean; 591 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: nvmeof daemon_check_post [(nvmeof.nvmeof.a), (nvmeof.nvmeof.b), (nvmeof.nvmeof.c), (nvmeof.nvmeof.d)] 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.a', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.a", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.b', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:25.851 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:25.852 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:25.852 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:25 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2436766328' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:26.057 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:26.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:26.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:26 smithi064 ceph-mon[17163]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.c', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:26.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:26 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:26.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:26 smithi064 ceph-mon[17163]: from='client.? 172.21.15.87:0/1896709486' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:27.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:26 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:27.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:26 smithi087 ceph-mon[21100]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.c', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:27.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:26 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:27.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:26 smithi087 ceph-mon[21100]: from='client.? 172.21.15.87:0/1896709486' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:27.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:27.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:26 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.b", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:27.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:26 smithi094 ceph-mon[20404]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.c', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:27.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:26 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:27.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:26 smithi094 ceph-mon[20404]: from='client.? 172.21.15.87:0/1896709486' entity='client.nvmeof.nvmeof.b' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:27.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:27.133 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage31 --size 8192 2024-09-02T12:01:27.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage31 --size 8192 2024-09-02T12:01:27.207 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:27.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:27.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:27.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:27.426 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:27.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:27.499 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:27.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:27.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:27.637 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:27.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:27 smithi064 ceph-mon[17163]: pgmap v235: 33 pgs: 33 active+clean; 592 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-02T12:01:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:27 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:27 smithi064 ceph-mon[17163]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.d', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:27 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:27.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:27 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/1211543980' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:28.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:27 smithi087 ceph-mon[21100]: pgmap v235: 33 pgs: 33 active+clean; 592 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-02T12:01:28.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:27 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:28.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:27 smithi087 ceph-mon[21100]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.d', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:28.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:27 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:28.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:27 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/1211543980' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:28.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:27 smithi094 ceph-mon[20404]: pgmap v235: 33 pgs: 33 active+clean; 592 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-02T12:01:28.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:27 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.c", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:28.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:27 smithi094 ceph-mon[20404]: create gateway: monitor command {'prefix': 'nvme-gw create', 'id': 'client.nvmeof.nvmeof.d', 'group': 'mygroup0', 'pool': 'mypool'} 2024-09-02T12:01:28.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:27 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"} : dispatch 2024-09-02T12:01:28.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:27 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/1211543980' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:28.500 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage32 --size 8192 2024-09-02T12:01:28.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage32 --size 8192 2024-09-02T12:01:28.751 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:29.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:29.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-list"} : dispatch 2024-09-02T12:01:29.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-list"}]: dispatch 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: Adding NVMeoF gateway 172.21.15.64:5500 to Dashboard 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: Adding NVMeoF gateway 172.21.15.87:5500 to Dashboard 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: Adding NVMeoF gateway 172.21.15.94:5500 to Dashboard 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: Adding NVMeoF gateway 172.21.15.99:5500 to Dashboard 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:01:29.002 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.003 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:28 smithi087 ceph-mon[21100]: from='client.? 172.21.15.99:0/2107647261' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:29.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:29.019 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-list"} : dispatch 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-list"}]: dispatch 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: Adding NVMeoF gateway 172.21.15.64:5500 to Dashboard 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: Adding NVMeoF gateway 172.21.15.87:5500 to Dashboard 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: Adding NVMeoF gateway 172.21.15.94:5500 to Dashboard 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: Adding NVMeoF gateway 172.21.15.99:5500 to Dashboard 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.020 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:01:29.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:29.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:01:29.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:28 smithi064 ceph-mon[17163]: from='client.? 172.21.15.99:0/2107647261' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:29.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd='[{"prefix": "nvme-gw create", "id": "client.nvmeof.nvmeof.d", "group": "mygroup0", "pool": "mypool"}]': finished 2024-09-02T12:01:29.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-list"} : dispatch 2024-09-02T12:01:29.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-list"}]: dispatch 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: Adding NVMeoF gateway 172.21.15.64:5500 to Dashboard 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: Adding NVMeoF gateway 172.21.15.87:5500 to Dashboard 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: Adding NVMeoF gateway 172.21.15.94:5500 to Dashboard 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: Adding NVMeoF gateway 172.21.15.99:5500 to Dashboard 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"} : dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard nvmeof-gateway-add", "name": "nvmeof.mypool.mygroup0"}]: dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:01:29.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:01:29.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:01:29.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:28 smithi094 ceph-mon[20404]: from='client.? 172.21.15.99:0/2107647261' entity='client.nvmeof.nvmeof.d' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:01:29.935 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage33 --size 8192 2024-09-02T12:01:29.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage33 --size 8192 2024-09-02T12:01:30.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:29 smithi064 ceph-mon[17163]: pgmap v236: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-02T12:01:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:29 smithi094 ceph-mon[20404]: pgmap v236: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-02T12:01:30.183 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:30.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:29 smithi087 ceph-mon[21100]: pgmap v236: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-02T12:01:31.458 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage34 --size 8192 2024-09-02T12:01:31.458 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage34 --size 8192 2024-09-02T12:01:31.703 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:32.052 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:31 smithi064 ceph-mon[17163]: pgmap v237: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-02T12:01:32.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:31 smithi094 ceph-mon[20404]: pgmap v237: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-02T12:01:32.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:31 smithi087 ceph-mon[21100]: pgmap v237: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-02T12:01:32.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:32.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:32.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:32.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:32.915 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:32.949 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage35 --size 8192 2024-09-02T12:01:32.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage35 --size 8192 2024-09-02T12:01:32.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:33.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:33.093 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:33.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:33.200 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:33.201 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:33.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:33.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:34.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:33 smithi064 ceph-mon[17163]: pgmap v238: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-02T12:01:34.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:33 smithi094 ceph-mon[20404]: pgmap v238: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-02T12:01:34.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:33 smithi087 ceph-mon[21100]: pgmap v238: 33 pgs: 33 active+clean; 594 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-02T12:01:34.358 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage36 --size 8192 2024-09-02T12:01:34.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage36 --size 8192 2024-09-02T12:01:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:35.758 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage37 --size 8192 2024-09-02T12:01:35.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage37 --size 8192 2024-09-02T12:01:36.006 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:35 smithi064 ceph-mon[17163]: pgmap v239: 33 pgs: 33 active+clean; 595 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-09-02T12:01:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:35 smithi094 ceph-mon[20404]: pgmap v239: 33 pgs: 33 active+clean; 595 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-09-02T12:01:36.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:35 smithi087 ceph-mon[21100]: pgmap v239: 33 pgs: 33 active+clean; 595 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 KiB/s wr, 8 op/s 2024-09-02T12:01:37.065 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage38 --size 8192 2024-09-02T12:01:37.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage38 --size 8192 2024-09-02T12:01:37.320 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:37 smithi064 ceph-mon[17163]: pgmap v240: 33 pgs: 33 active+clean; 596 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-09-02T12:01:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:37 smithi094 ceph-mon[20404]: pgmap v240: 33 pgs: 33 active+clean; 596 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-09-02T12:01:38.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:37 smithi087 ceph-mon[21100]: pgmap v240: 33 pgs: 33 active+clean; 596 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-09-02T12:01:38.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:38.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:38.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:38.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:38.514 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:38.587 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:38.619 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:38.640 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage39 --size 8192 2024-09-02T12:01:38.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage39 --size 8192 2024-09-02T12:01:38.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:38.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:38.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:38.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:38.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:38.897 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:40.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:39 smithi064 ceph-mon[17163]: pgmap v241: 33 pgs: 33 active+clean; 596 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-09-02T12:01:40.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:39 smithi094 ceph-mon[20404]: pgmap v241: 33 pgs: 33 active+clean; 596 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-09-02T12:01:40.158 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage40 --size 8192 2024-09-02T12:01:40.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage40 --size 8192 2024-09-02T12:01:40.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:39 smithi087 ceph-mon[21100]: pgmap v241: 33 pgs: 33 active+clean; 596 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-09-02T12:01:40.409 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:41.559 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage41 --size 8192 2024-09-02T12:01:41.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage41 --size 8192 2024-09-02T12:01:41.810 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:42.059 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:41 smithi064 ceph-mon[17163]: pgmap v242: 33 pgs: 33 active+clean; 597 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-02T12:01:42.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:41 smithi094 ceph-mon[20404]: pgmap v242: 33 pgs: 33 active+clean; 597 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-02T12:01:42.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:41 smithi087 ceph-mon[21100]: pgmap v242: 33 pgs: 33 active+clean; 597 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-02T12:01:43.074 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage42 --size 8192 2024-09-02T12:01:43.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage42 --size 8192 2024-09-02T12:01:43.320 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:43.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:43.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:43.994 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:43 smithi064 ceph-mon[17163]: pgmap v243: 33 pgs: 33 active+clean; 598 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-09-02T12:01:44.004 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:44.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:44.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:43 smithi094 ceph-mon[20404]: pgmap v243: 33 pgs: 33 active+clean; 598 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-09-02T12:01:44.097 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:43 smithi087 ceph-mon[21100]: pgmap v243: 33 pgs: 33 active+clean; 598 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-09-02T12:01:44.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:44.181 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:44.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:44.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:44.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:44.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:44.421 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:44.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:44.477 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage43 --size 8192 2024-09-02T12:01:44.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage43 --size 8192 2024-09-02T12:01:44.735 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:45.889 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage44 --size 8192 2024-09-02T12:01:45.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage44 --size 8192 2024-09-02T12:01:46.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:45 smithi064 ceph-mon[17163]: pgmap v244: 33 pgs: 33 active+clean; 599 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-09-02T12:01:46.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:45 smithi094 ceph-mon[20404]: pgmap v244: 33 pgs: 33 active+clean; 599 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-09-02T12:01:46.135 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:46.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:45 smithi087 ceph-mon[21100]: pgmap v244: 33 pgs: 33 active+clean; 599 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-09-02T12:01:47.733 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage45 --size 8192 2024-09-02T12:01:47.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage45 --size 8192 2024-09-02T12:01:47.984 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:47 smithi064 ceph-mon[17163]: pgmap v245: 33 pgs: 33 active+clean; 600 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:01:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:47 smithi094 ceph-mon[20404]: pgmap v245: 33 pgs: 33 active+clean; 600 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:01:48.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:47 smithi087 ceph-mon[21100]: pgmap v245: 33 pgs: 33 active+clean; 600 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:01:49.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:49.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:49.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:49.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:49.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:49.769 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:49.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:49.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:49.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:49.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:49.999 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:49 smithi087 ceph-mon[21100]: pgmap v246: 33 pgs: 33 active+clean; 600 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-09-02T12:01:50.022 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:50.043 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:49 smithi094 ceph-mon[20404]: pgmap v246: 33 pgs: 33 active+clean; 600 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-09-02T12:01:50.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:50.075 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:49 smithi064 ceph-mon[17163]: pgmap v246: 33 pgs: 33 active+clean; 600 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-09-02T12:01:52.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:51 smithi064 ceph-mon[17163]: pgmap v247: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-02T12:01:52.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:51 smithi094 ceph-mon[20404]: pgmap v247: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-02T12:01:52.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:51 smithi087 ceph-mon[21100]: pgmap v247: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-02T12:01:54.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:53 smithi064 ceph-mon[17163]: pgmap v248: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-09-02T12:01:54.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:53 smithi094 ceph-mon[20404]: pgmap v248: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-09-02T12:01:54.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:53 smithi087 ceph-mon[21100]: pgmap v248: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-09-02T12:01:54.435 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage46 --size 8192 2024-09-02T12:01:54.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage46 --size 8192 2024-09-02T12:01:54.684 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:55.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:01:55.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:01:55.167 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:01:55.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:01:55.284 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:01:55.355 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:01:55.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:01:55.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:01:55.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:01:55.561 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:01:55.593 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:01:55.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:01:55.959 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage47 --size 8192 2024-09-02T12:01:55.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage47 --size 8192 2024-09-02T12:01:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:55 smithi087 ceph-mon[21100]: pgmap v249: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-02T12:01:56.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:55 smithi064 ceph-mon[17163]: pgmap v249: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-02T12:01:56.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:55 smithi094 ceph-mon[20404]: pgmap v249: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-02T12:01:56.212 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:57.333 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage48 --size 8192 2024-09-02T12:01:57.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage48 --size 8192 2024-09-02T12:01:57.579 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:01:57.954 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:57 smithi064 ceph-mon[17163]: pgmap v250: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-02T12:01:58.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:57 smithi094 ceph-mon[20404]: pgmap v250: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-02T12:01:58.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:57 smithi087 ceph-mon[21100]: pgmap v250: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-09-02T12:01:58.624 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage49 --size 8192 2024-09-02T12:01:58.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage49 --size 8192 2024-09-02T12:01:58.869 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:01:59 smithi064 ceph-mon[17163]: pgmap v251: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-02T12:02:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:01:59 smithi094 ceph-mon[20404]: pgmap v251: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-02T12:02:00.165 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage50 --size 8192 2024-09-02T12:02:00.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage50 --size 8192 2024-09-02T12:02:00.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:01:59 smithi087 ceph-mon[21100]: pgmap v251: 33 pgs: 33 active+clean; 601 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-02T12:02:00.412 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:00.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:00.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:00.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:00.795 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:00.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:00.939 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:00.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:01.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:01.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:01.126 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:01.159 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:01.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:01.723 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage51 --size 8192 2024-09-02T12:02:01.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage51 --size 8192 2024-09-02T12:02:01.971 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:02.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:01 smithi064 ceph-mon[17163]: pgmap v252: 33 pgs: 33 active+clean; 603 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-02T12:02:02.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:01 smithi094 ceph-mon[20404]: pgmap v252: 33 pgs: 33 active+clean; 603 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-02T12:02:02.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:01 smithi087 ceph-mon[21100]: pgmap v252: 33 pgs: 33 active+clean; 603 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-02T12:02:03.221 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage52 --size 8192 2024-09-02T12:02:03.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage52 --size 8192 2024-09-02T12:02:03.468 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:03.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:03 smithi064 ceph-mon[17163]: pgmap v253: 33 pgs: 33 active+clean; 603 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-09-02T12:02:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:03 smithi094 ceph-mon[20404]: pgmap v253: 33 pgs: 33 active+clean; 603 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-09-02T12:02:04.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:03 smithi087 ceph-mon[21100]: pgmap v253: 33 pgs: 33 active+clean; 603 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-09-02T12:02:04.653 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage53 --size 8192 2024-09-02T12:02:04.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage53 --size 8192 2024-09-02T12:02:04.905 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:05.987 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage54 --size 8192 2024-09-02T12:02:05.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage54 --size 8192 2024-09-02T12:02:06.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:05 smithi064 ceph-mon[17163]: pgmap v254: 33 pgs: 33 active+clean; 604 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-09-02T12:02:06.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:05 smithi094 ceph-mon[20404]: pgmap v254: 33 pgs: 33 active+clean; 604 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-09-02T12:02:06.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:06.241 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:06.249 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:05 smithi087 ceph-mon[21100]: pgmap v254: 33 pgs: 33 active+clean; 604 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-09-02T12:02:06.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:06.333 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:06.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:06.438 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:06.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:06.544 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:06.617 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:06.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:06.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:06.754 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:06.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:07.536 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage55 --size 8192 2024-09-02T12:02:07.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage55 --size 8192 2024-09-02T12:02:07.790 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:08.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:07 smithi064 ceph-mon[17163]: pgmap v255: 33 pgs: 33 active+clean; 605 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-09-02T12:02:08.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:07 smithi094 ceph-mon[20404]: pgmap v255: 33 pgs: 33 active+clean; 605 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-09-02T12:02:08.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:07 smithi087 ceph-mon[21100]: pgmap v255: 33 pgs: 33 active+clean; 605 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-09-02T12:02:09.004 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage56 --size 8192 2024-09-02T12:02:09.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage56 --size 8192 2024-09-02T12:02:09.268 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:10.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:09 smithi064 ceph-mon[17163]: pgmap v256: 33 pgs: 33 active+clean; 606 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:10.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:09 smithi094 ceph-mon[20404]: pgmap v256: 33 pgs: 33 active+clean; 606 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:10.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:09 smithi087 ceph-mon[21100]: pgmap v256: 33 pgs: 33 active+clean; 606 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:10.511 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage57 --size 8192 2024-09-02T12:02:10.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage57 --size 8192 2024-09-02T12:02:10.770 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:11.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:11.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:11.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:11.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:11.976 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:11 smithi064 ceph-mon[17163]: pgmap v257: 33 pgs: 33 active+clean; 606 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:02:12.001 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage58 --size 8192 2024-09-02T12:02:12.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage58 --size 8192 2024-09-02T12:02:12.025 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:12.086 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:11 smithi087 ceph-mon[21100]: pgmap v257: 33 pgs: 33 active+clean; 606 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:02:12.096 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:12.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:11 smithi094 ceph-mon[20404]: pgmap v257: 33 pgs: 33 active+clean; 606 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:02:12.129 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:12.201 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:12.247 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:12.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:12.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:12.337 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:12.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:13.376 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage59 --size 8192 2024-09-02T12:02:13.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage59 --size 8192 2024-09-02T12:02:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:14.031 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:13 smithi064 ceph-mon[17163]: pgmap v258: 33 pgs: 33 active+clean; 607 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-09-02T12:02:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:13 smithi094 ceph-mon[20404]: pgmap v258: 33 pgs: 33 active+clean; 607 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-09-02T12:02:14.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:13 smithi087 ceph-mon[21100]: pgmap v258: 33 pgs: 33 active+clean; 607 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-09-02T12:02:14.791 INFO:tasks.nvmeof:[nvmeof]: rbd create mypool/myimage60 --size 8192 2024-09-02T12:02:14.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- rbd create mypool/myimage60 --size 8192 2024-09-02T12:02:15.038 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:16.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:15 smithi064 ceph-mon[17163]: pgmap v259: 33 pgs: 33 active+clean; 607 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:16.095 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:15 smithi094 ceph-mon[20404]: pgmap v259: 33 pgs: 33 active+clean; 607 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:16.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2024-09-02T12:02:16.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:15 smithi087 ceph-mon[21100]: pgmap v259: 33 pgs: 33 active+clean; 607 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:16.463 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:17.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:17.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:17.505 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:17.538 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:17.610 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:17.682 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:17.715 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:17.787 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:17.796 DEBUG:teuthology.orchestra.run.smithi064:nvmeof.nvmeof.a> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service 2024-09-02T12:02:17.845 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2024-09-02T12:02:17.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:17.916 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:17.950 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:17.970 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:17 smithi094 ceph-mon[20404]: pgmap v260: 33 pgs: 33 active+clean; 608 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:02:17.971 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:17 smithi094 ceph-mon[20404]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:17.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:18.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:17 smithi064 ceph-mon[17163]: pgmap v260: 33 pgs: 33 active+clean; 608 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:02:18.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:17 smithi064 ceph-mon[17163]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T12:02:18.217 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:17 smithi087 ceph-mon[21100]: pgmap v260: 33 pgs: 33 active+clean; 608 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-02T12:02:18.217 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:17 smithi087 ceph-mon[21100]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:19.660 DEBUG:teuthology.orchestra.run.smithi087:nvmeof.nvmeof.b> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service 2024-09-02T12:02:19.662 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2024-09-02T12:02:19.909 INFO:teuthology.orchestra.run.smithi094.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T12:02:20.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:19 smithi087 ceph-mon[21100]: pgmap v261: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:20.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:19 smithi087 ceph-mon[21100]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:20.029 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:19 smithi094 ceph-mon[20404]: pgmap v261: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:20.029 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:19 smithi094 ceph-mon[20404]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:20.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:19 smithi064 ceph-mon[17163]: pgmap v261: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-09-02T12:02:20.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:19 smithi064 ceph-mon[17163]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:21.512 DEBUG:teuthology.orchestra.run.smithi094:nvmeof.nvmeof.c> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service 2024-09-02T12:02:21.515 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch ls nvmeof --export > /etc/ceph/nvmeof-gw.yaml 2024-09-02T12:02:21.761 INFO:teuthology.orchestra.run.smithi099.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T12:02:22.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:21 smithi064 ceph-mon[17163]: pgmap v262: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-02T12:02:22.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:21 smithi064 ceph-mon[17163]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:22.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:21 smithi094 ceph-mon[20404]: pgmap v262: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-02T12:02:22.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:21 smithi094 ceph-mon[20404]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:22.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:21 smithi087 ceph-mon[21100]: pgmap v262: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-02T12:02:22.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:21 smithi087 ceph-mon[21100]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:23.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:23.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:23.115 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:23.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:23.224 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:23.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:23.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:23.331 DEBUG:teuthology.orchestra.run.smithi099:nvmeof.nvmeof.d> sudo journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service 2024-09-02T12:02:23.333 INFO:tasks.nvmeof:[nvmeof]: executed deploy_nvmeof successfully! 2024-09-02T12:02:23.333 INFO:tasks.nvmeof:[nvmeof]: running set_gateway_cfg... 2024-09-02T12:02:23.334 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T12:02:23.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/nvmeof.env 2024-09-02T12:02:23.367 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-02T12:02:23.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/nvmeof.env 2024-09-02T12:02:23.392 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-02T12:02:23.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/nvmeof.env 2024-09-02T12:02:23.401 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:23.417 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T12:02:23.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/nvmeof.env 2024-09-02T12:02:23.466 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-02T12:02:23.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/nvmeof.env 2024-09-02T12:02:23.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:23.491 INFO:tasks.nvmeof:[nvmeof]: executed set_gateway_cfg successfully! 2024-09-02T12:02:23.491 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-09-02T12:02:23.502 INFO:tasks.cephadm:Waiting for ceph service nvmeof.mypool.mygroup0 to start (timeout 300)... 2024-09-02T12:02:23.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8e22764-6921-11ef-bcd4-c7b262605968 -- ceph orch ls -f json 2024-09-02T12:02:23.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:23.537 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:23.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:23.794 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T12:02:24.192 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:23 smithi064 ceph-mon[17163]: pgmap v263: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-09-02T12:02:24.192 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:23 smithi064 ceph-mon[17163]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:24.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:23 smithi087 ceph-mon[21100]: pgmap v263: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-09-02T12:02:24.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:23 smithi087 ceph-mon[21100]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:24.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:23 smithi094 ceph-mon[20404]: pgmap v263: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-09-02T12:02:24.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:23 smithi094 ceph-mon[20404]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch ls", "service_type": "nvmeof", "export": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:02:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T12:02:24.478 INFO:teuthology.orchestra.run.smithi064.stdout:[{"events": ["2024-09-02T11:56:16.251966Z service:mgr [INFO] \"service was created\""], "placement": {"count": 1, "hosts": ["smithi064=x"]}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-09-02T11:56:15.971826Z", "last_refresh": "2024-09-02T12:01:24.505667Z", "running": 1, "size": 1}}, {"events": ["2024-09-02T11:56:09.960437Z service:mon [INFO] \"service was created\""], "placement": {"count": 3, "hosts": ["smithi064:172.21.15.64=a", "smithi087:172.21.15.87=b", "smithi094:172.21.15.94=c"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-09-02T11:55:51.733947Z", "last_refresh": "2024-09-02T12:01:23.118509Z", "running": 3, "size": 3}}, {"events": ["2024-09-02T12:01:21.718604Z service:nvmeof.mypool.mygroup0 [INFO] \"service was created\""], "placement": {"hosts": ["smithi064=nvmeof.a", "smithi087=nvmeof.b", "smithi094=nvmeof.c", "smithi099=nvmeof.d"]}, "service_id": "mypool.mygroup0", "service_name": "nvmeof.mypool.mygroup0", "service_type": "nvmeof", "spec": {"allowed_consecutive_spdk_ping_failures": 1, "bdevs_per_cluster": 32, "conn_retries": 10, "discovery_port": 8009, "enable_monitor_client": true, "enable_prometheus_exporter": true, "group": "mygroup0", "log_directory": "/var/log/ceph/", "log_files_enabled": true, "log_files_rotation_enabled": true, "log_level": "INFO", "max_log_directory_backups": 10, "max_log_file_size_in_mb": 10, "max_log_files_count": 20, "monitor_timeout": 1.0, "omap_file_lock_duration": 20, "omap_file_lock_retries": 30, "omap_file_lock_retry_sleep_interval": 1.0, "omap_file_update_reloads": 10, "pool": "mypool", "port": 5500, "rpc_socket": "/var/tmp/spdk.sock", "spdk_path": "/usr/local/bin/nvmf_tgt", "spdk_ping_interval_in_seconds": 2.0, "spdk_protocol_log_level": "WARNING", "spdk_timeout": 60.0, "state_update_interval_sec": 5, "state_update_notify": true, "tgt_path": "/usr/local/bin/nvmf_tgt", "transport_tcp_options": {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, "transports": "tcp", "verbose_log_messages": true, "verify_nqns": true}, "status": {"created": "2024-09-02T12:00:35.990462Z", "last_refresh": "2024-09-02T12:01:22.440044Z", "ports": [5500, 4420, 8009], "running": 4, "size": 4}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238", "container_image_name": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81", "last_refresh": "2024-09-02T12:01:23.118601Z", "running": 8, "size": 8}, "unmanaged": true}] 2024-09-02T12:02:25.248 INFO:tasks.cephadm:nvmeof.mypool.mygroup0 has 4/4 2024-09-02T12:02:25.248 INFO:teuthology.run_tasks:Running task workunit... 2024-09-02T12:02:25.263 INFO:tasks.workunit:Pulling workunits from ref 6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 2024-09-02T12:02:25.264 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-02T12:02:25.264 DEBUG:teuthology.orchestra.run.smithi038:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-02T12:02:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:02:25.283 INFO:teuthology.orchestra.run.smithi038.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-02T12:02:25.283 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-02T12:02:25.340 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-02T12:02:25.341 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-02T12:02:25.398 INFO:tasks.workunit:timeout=3h 2024-09-02T12:02:25.398 INFO:tasks.workunit:cleanup=True 2024-09-02T12:02:25.398 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/vallariag/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 2024-09-02T12:02:25.455 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-02T12:02:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:25 smithi087 ceph-mon[21100]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:02:26.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:25 smithi087 ceph-mon[21100]: pgmap v264: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-02T12:02:26.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:25 smithi064 ceph-mon[17163]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:02:26.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:25 smithi064 ceph-mon[17163]: pgmap v264: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-02T12:02:26.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:25 smithi094 ceph-mon[20404]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T12:02:26.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:25 smithi094 ceph-mon[20404]: pgmap v264: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-09-02T12:02:28.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:27 smithi087 ceph-mon[21100]: pgmap v265: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-09-02T12:02:28.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:27 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:02:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:27 smithi064 ceph-mon[17163]: pgmap v265: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-09-02T12:02:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:27 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:02:28.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:27 smithi094 ceph-mon[20404]: pgmap v265: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 1023 B/s wr, 7 op/s 2024-09-02T12:02:28.349 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:27 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:02:28.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:28.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:28.721 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:28.754 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:28.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:28.898 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:28.930 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:29.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:29.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:29.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:29.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:29.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:29.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:02:29.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:02:29.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:28 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:02:29.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:02:29.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:02:29.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:28 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:02:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:02:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:02:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:28 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:02:30.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:29 smithi087 ceph-mon[21100]: pgmap v266: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 597 B/s wr, 4 op/s 2024-09-02T12:02:30.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:29 smithi064 ceph-mon[17163]: pgmap v266: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 597 B/s wr, 4 op/s 2024-09-02T12:02:30.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:29 smithi094 ceph-mon[20404]: pgmap v266: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 597 B/s wr, 4 op/s 2024-09-02T12:02:32.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:31 smithi087 ceph-mon[21100]: pgmap v267: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-02T12:02:32.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:31 smithi064 ceph-mon[17163]: pgmap v267: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-02T12:02:32.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:31 smithi094 ceph-mon[20404]: pgmap v267: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-02T12:02:34.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:34.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:34.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:33 smithi087 ceph-mon[21100]: pgmap v268: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:34.289 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:33 smithi064 ceph-mon[17163]: pgmap v268: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:34.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:34.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:34.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:33 smithi094 ceph-mon[20404]: pgmap v268: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:34.404 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:34.475 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:34.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:34.577 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:34.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:34.679 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:34.710 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:34.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:35 smithi064 ceph-mon[17163]: pgmap v269: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:36.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:35 smithi094 ceph-mon[20404]: pgmap v269: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:36.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:36 smithi087 ceph-mon[21100]: pgmap v269: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:38.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:38 smithi064 ceph-mon[17163]: pgmap v270: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:38 smithi094 ceph-mon[20404]: pgmap v270: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:38.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:38 smithi087 ceph-mon[21100]: pgmap v270: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:39.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:39.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:39.875 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:39.908 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:39.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:40.050 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:40.081 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:40.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:40 smithi064 ceph-mon[17163]: pgmap v271: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:40.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:40.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:40.249 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:40 smithi087 ceph-mon[21100]: pgmap v271: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:40.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:40.286 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:40.307 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:40 smithi094 ceph-mon[20404]: pgmap v271: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:40.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:42.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:42 smithi064 ceph-mon[17163]: pgmap v272: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:42.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:42 smithi094 ceph-mon[20404]: pgmap v272: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:42 smithi087 ceph-mon[21100]: pgmap v272: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:44.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:44 smithi064 ceph-mon[17163]: pgmap v273: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:44.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:44 smithi094 ceph-mon[20404]: pgmap v273: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:44 smithi087 ceph-mon[21100]: pgmap v273: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:45.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:45.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:45.452 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:45.485 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:45.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:45.631 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:45.662 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:45.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:45.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:45.834 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:45.867 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:45.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:46.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:46 smithi064 ceph-mon[17163]: pgmap v274: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:46.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:46 smithi087 ceph-mon[21100]: pgmap v274: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:46.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:46 smithi094 ceph-mon[20404]: pgmap v274: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:48.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:48 smithi064 ceph-mon[17163]: pgmap v275: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:48.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:48 smithi094 ceph-mon[20404]: pgmap v275: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:48.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:48 smithi087 ceph-mon[21100]: pgmap v275: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:50.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:50 smithi064 ceph-mon[17163]: pgmap v276: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:50.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:50 smithi094 ceph-mon[20404]: pgmap v276: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:50.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:50 smithi087 ceph-mon[21100]: pgmap v276: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:50.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:50.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:51.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:51.065 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:51.136 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:51.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:51.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:51.308 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:51.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:51.410 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:51.442 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:51.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:52.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:52 smithi064 ceph-mon[17163]: pgmap v277: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:52.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:52 smithi094 ceph-mon[20404]: pgmap v277: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:52.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:52 smithi087 ceph-mon[21100]: pgmap v277: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:54.302 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:54 smithi087 ceph-mon[21100]: pgmap v278: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:54.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:54 smithi064 ceph-mon[17163]: pgmap v278: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:54.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:54 smithi094 ceph-mon[20404]: pgmap v278: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:56.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:56 smithi064 ceph-mon[17163]: pgmap v279: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:56.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:56 smithi094 ceph-mon[20404]: pgmap v279: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:56.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:56 smithi087 ceph-mon[21100]: pgmap v279: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:02:56.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:02:56.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:02:56.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:02:56.636 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:02:56.708 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:02:56.779 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:02:56.813 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:02:56.884 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:02:56.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:02:56.985 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:02:57.017 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:02:57.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:02:58.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:02:58 smithi064 ceph-mon[17163]: pgmap v280: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:58.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:02:58 smithi094 ceph-mon[20404]: pgmap v280: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:02:58.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:02:58 smithi087 ceph-mon[21100]: pgmap v280: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:00.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:00 smithi064 ceph-mon[17163]: pgmap v281: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:00.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:00 smithi094 ceph-mon[20404]: pgmap v281: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:00.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:00 smithi087 ceph-mon[21100]: pgmap v281: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:02.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:02.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:02.180 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:02.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:02.273 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:02 smithi087 ceph-mon[21100]: pgmap v282: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:02.284 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:02 smithi064 ceph-mon[17163]: pgmap v282: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:02.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:02 smithi094 ceph-mon[20404]: pgmap v282: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:02.356 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:02.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:02.458 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:02.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:02.561 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:02.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:02.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:04.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:04 smithi064 ceph-mon[17163]: pgmap v283: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:04.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:04 smithi094 ceph-mon[20404]: pgmap v283: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:04.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:04 smithi087 ceph-mon[21100]: pgmap v283: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:05.678 INFO:tasks.workunit.client.0.smithi038.stderr:Note: switching to '6bdd3521695a0621868ce5034eaba1fd0b6e3cd1'. 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr:state without impacting any branches by switching back to a branch. 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.679 INFO:tasks.workunit.client.0.smithi038.stderr: git switch -c 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr:Or undo this operation with: 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr: git switch - 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-09-02T12:03:05.680 INFO:tasks.workunit.client.0.smithi038.stderr:HEAD is now at 6bdd3521695 qa/suites/nvmeof: wait for service "nvmeof.mypool.mygroup0" 2024-09-02T12:03:05.683 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-02T12:03:05.746 INFO:tasks.workunit.client.0.smithi038.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-02T12:03:05.749 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-02T12:03:05.749 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-02T12:03:06.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:06 smithi064 ceph-mon[17163]: pgmap v284: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:06.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:06 smithi094 ceph-mon[20404]: pgmap v284: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:06.352 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-02T12:03:06.414 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-02T12:03:06.453 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-02T12:03:06.454 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-02T12:03:06.455 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-02T12:03:06.493 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-02T12:03:06.497 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-02T12:03:06.497 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-02T12:03:06.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:06 smithi087 ceph-mon[21100]: pgmap v284: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:06.552 INFO:tasks.workunit:Running workunits matching nvmeof/setup_subsystem.sh on client.0... 2024-09-02T12:03:06.553 INFO:tasks.workunit:Running workunit nvmeof/setup_subsystem.sh... 2024-09-02T12:03:06.554 DEBUG:teuthology.orchestra.run.smithi038:workunit test nvmeof/setup_subsystem.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_IMAGE_PREFIX=myimage RBD_POOL=mypool timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/setup_subsystem.sh 2024-09-02T12:03:06.609 INFO:tasks.workunit.client.0.smithi038.stderr:+ source /etc/ceph/nvmeof.env 2024-09-02T12:03:06.609 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_GATEWAY_IP_ADDRESSES=172.21.15.64,172.21.15.87,172.21.15.94,172.21.15.99 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_GATEWAY_NAMES=smithi064,smithi087,smithi094,smithi099 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS=172.21.15.64 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_CLI_IMAGE=quay.io/ceph/nvmeof-cli:1.2 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_SUBSYSTEMS_PREFIX=nqn.2016-06.io.spdk:cnode 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_SUBSYSTEMS_COUNT=3 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_NAMESPACES_COUNT=20 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_PORT=4420 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_SRPORT=5500 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:+ RBD_POOL=mypool 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:+ RBD_IMAGE_PREFIX=myimage 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:++ hostname 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:+ HOSTNAME=smithi038 2024-09-02T12:03:06.610 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman images 2024-09-02T12:03:06.639 INFO:tasks.workunit.client.0.smithi038.stdout:REPOSITORY TAG IMAGE ID CREATED SIZE 2024-09-02T12:03:06.639 INFO:tasks.workunit.client.0.smithi038.stdout:quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph b59673c44bd569f9f3db37f87bced695dec5fcbf b92d9064031c 4 days ago 1.43 GB 2024-09-02T12:03:06.642 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman ps 2024-09-02T12:03:06.669 INFO:tasks.workunit.client.0.smithi038.stdout:CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2024-09-02T12:03:06.672 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 --format json subsystem list 2024-09-02T12:03:06.700 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:06Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:06.783 INFO:tasks.workunit.client.0.smithi038.stderr:Trying to pull quay.io/ceph/nvmeof-cli:1.2... 2024-09-02T12:03:07.178 INFO:tasks.workunit.client.0.smithi038.stderr:Getting image source signatures 2024-09-02T12:03:07.178 INFO:tasks.workunit.client.0.smithi038.stderr:Copying blob sha256:081ceb2853645ae0182f63522de488e2204b2cc27d4f897a82bd7a428233a198 2024-09-02T12:03:07.178 INFO:tasks.workunit.client.0.smithi038.stderr:Copying blob sha256:011553e07ff8c59aadf26951274ff47cec461fc15a46d4e56e794afc4b2ed4fd 2024-09-02T12:03:07.178 INFO:tasks.workunit.client.0.smithi038.stderr:Copying blob sha256:a52d3e976d4d1da9959dd280155c706b6bb2460ca8b01567798318b23ec12b43 2024-09-02T12:03:07.178 INFO:tasks.workunit.client.0.smithi038.stderr:Copying blob sha256:99a7ec374c4d4af5f9b9ad281068affa7961aad432aba4efc963785a5a350f8d 2024-09-02T12:03:07.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:07.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:07.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:07.789 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:07.862 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:07.935 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:07.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:08.037 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:08.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:08.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:08.171 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:08.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:08.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:08 smithi064 ceph-mon[17163]: pgmap v285: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:08.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:08 smithi094 ceph-mon[20404]: pgmap v285: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:08.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:08 smithi087 ceph-mon[21100]: pgmap v285: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:10.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:10 smithi064 ceph-mon[17163]: pgmap v286: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:10 smithi094 ceph-mon[20404]: pgmap v286: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:10.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:10 smithi087 ceph-mon[21100]: pgmap v286: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:11.937 INFO:tasks.workunit.client.0.smithi038.stderr:Copying config sha256:f51c0db7f067fe1683795c59dd8c032b14c21a84238f69f1d632cedf9f5832f6 2024-09-02T12:03:12.096 INFO:tasks.workunit.client.0.smithi038.stderr:Writing manifest to image destination 2024-09-02T12:03:12.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:12 smithi064 ceph-mon[17163]: pgmap v287: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:12.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:12 smithi094 ceph-mon[20404]: pgmap v287: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:12 smithi087 ceph-mon[21100]: pgmap v287: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:13.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:13.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:13.338 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:13.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:13.443 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:13.514 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:13.526 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:03:13.526 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:03:13.526 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0, 2024-09-02T12:03:13.526 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [] 2024-09-02T12:03:13.526 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:03:13.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:13.615 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:13.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:13.708 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:13] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:39862 2024-09-02T12:03:13.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:13.748 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:13.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:14.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:14 smithi064 ceph-mon[17163]: pgmap v288: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:14.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:14 smithi094 ceph-mon[20404]: pgmap v288: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:14.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:14 smithi087 ceph-mon[21100]: pgmap v288: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:16.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:16 smithi064 ceph-mon[17163]: pgmap v289: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:16.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:16 smithi094 ceph-mon[20404]: pgmap v289: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:16 smithi087 ceph-mon[21100]: pgmap v289: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:18.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:18 smithi064 ceph-mon[17163]: pgmap v290: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:18.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:18 smithi094 ceph-mon[20404]: pgmap v290: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:18.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:18 smithi087 ceph-mon[21100]: pgmap v290: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:18.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:18.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:18.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:18.945 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:19.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:19.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:19.125 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:19.196 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:19.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:19.298 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:19.331 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:19.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:19.855 INFO:tasks.workunit.client.0.smithi038.stderr:+ IFS=, 2024-09-02T12:03:19.855 INFO:tasks.workunit.client.0.smithi038.stderr:+ read -ra gateway_ips 2024-09-02T12:03:19.855 INFO:tasks.workunit.client.0.smithi038.stderr:+ IFS=, 2024-09-02T12:03:19.855 INFO:tasks.workunit.client.0.smithi038.stderr:+ read -ra gateway_names 2024-09-02T12:03:19.856 INFO:tasks.workunit.client.0.smithi038.stderr:++ echo 172.21.15.64,172.21.15.87,172.21.15.94,172.21.15.99 2024-09-02T12:03:19.856 INFO:tasks.workunit.client.0.smithi038.stderr:++ tr -cd , 2024-09-02T12:03:19.856 INFO:tasks.workunit.client.0.smithi038.stderr:++ wc -c 2024-09-02T12:03:19.861 INFO:tasks.workunit.client.0.smithi038.stderr:+ gateways_count=4 2024-09-02T12:03:19.862 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:19.866 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:19.867 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:19.867 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:19.940 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:20.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:20 smithi064 ceph-mon[17163]: pgmap v291: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:20.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:20 smithi094 ceph-mon[20404]: pgmap v291: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:20.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:20 smithi087 ceph-mon[21100]: pgmap v291: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:20.684 INFO:tasks.workunit.client.0.smithi038.stdout:Adding subsystem nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:03:20.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:20] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:21.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:20] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:21.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:20] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 256, context: , client address: IPv4 172.21.15.38:53032 2024-09-02T12:03:21.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:20] INFO grpc.py:545 (2): No serial number specified for nqn.2016-06.io.spdk:cnode1, will use Ceph81601849504533 2024-09-02T12:03:21.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:20] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:21.565 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:21.565 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:21.565 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:22.271 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:22Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:22.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:22 smithi064 ceph-mon[17163]: pgmap v292: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:22.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-mon[20404]: pgmap v292: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:22.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:22] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:03:22.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: states { 2024-09-02T12:03:22.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: grp_id: 3 2024-09-02T12:03:22.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: state: OPTIMIZED 2024-09-02T12:03:22.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: } 2024-09-02T12:03:22.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: ], , client address: IPv4 172.21.15.94:52274 2024-09-02T12:03:22.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:22] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:03:22.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: states { 2024-09-02T12:03:22.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: grp_id: 4 2024-09-02T12:03:22.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: state: OPTIMIZED 2024-09-02T12:03:22.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: } 2024-09-02T12:03:22.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: ], , client address: IPv4 172.21.15.99:37626 2024-09-02T12:03:22.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-mon[21100]: pgmap v292: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 3 op/s 2024-09-02T12:03:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:22] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:03:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: states { 2024-09-02T12:03:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: grp_id: 2 2024-09-02T12:03:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: state: OPTIMIZED 2024-09-02T12:03:22.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: } 2024-09-02T12:03:22.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: ], , client address: IPv4 172.21.15.87:42242 2024-09-02T12:03:23.031 INFO:tasks.workunit.client.0.smithi038.stdout:Adding subsystem nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:03:23.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:23] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 256, context: , client address: IPv4 172.21.15.38:35740 2024-09-02T12:03:23.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:23] INFO grpc.py:545 (2): No serial number specified for nqn.2016-06.io.spdk:cnode2, will use Ceph61660518994858 2024-09-02T12:03:23.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:23] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:03:23.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:03:23.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 1 2024-09-02T12:03:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:03:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:03:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:36242 2024-09-02T12:03:23.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:23] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:23.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:23] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:23.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:23] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:23.892 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:23.892 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:23.892 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:23.977 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:23Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:24.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:24 smithi064 ceph-mon[17163]: pgmap v293: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:24.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-mon[20404]: pgmap v293: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:24.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:24] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:03:24.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: states { 2024-09-02T12:03:24.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: grp_id: 3 2024-09-02T12:03:24.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: state: OPTIMIZED 2024-09-02T12:03:24.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: } 2024-09-02T12:03:24.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: ], , client address: IPv4 172.21.15.94:52286 2024-09-02T12:03:24.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:24.401 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-mon[21100]: pgmap v293: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:24.401 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:24] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:03:24.401 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: states { 2024-09-02T12:03:24.401 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: grp_id: 2 2024-09-02T12:03:24.401 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: state: OPTIMIZED 2024-09-02T12:03:24.401 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: } 2024-09-02T12:03:24.401 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: ], , client address: IPv4 172.21.15.87:42256 2024-09-02T12:03:24.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:24.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:24] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:03:24.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: states { 2024-09-02T12:03:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: grp_id: 4 2024-09-02T12:03:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: state: OPTIMIZED 2024-09-02T12:03:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: } 2024-09-02T12:03:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: ], , client address: IPv4 172.21.15.99:37632 2024-09-02T12:03:24.495 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:24.529 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:24.601 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:24.674 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:24.707 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:24.735 INFO:tasks.workunit.client.0.smithi038.stdout:Adding subsystem nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:03:24.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:24.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:24.879 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:24.911 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:24.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:24.999 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:24 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:24] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:25.056 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:24 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:24] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 256, context: , client address: IPv4 172.21.15.38:34432 2024-09-02T12:03:25.057 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:24 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:24] INFO grpc.py:545 (2): No serial number specified for nqn.2016-06.io.spdk:cnode3, will use Ceph22728624249465 2024-09-02T12:03:25.057 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:25] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:03:25.057 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:03:25.057 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 1 2024-09-02T12:03:25.057 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:03:25.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:24 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:24] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:25.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:24] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:03:25.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:03:25.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:36260 2024-09-02T12:03:25.596 INFO:tasks.workunit.client.0.smithi038.stderr:+ list_subsystems 2024-09-02T12:03:25.596 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:25.596 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.64 2024-09-02T12:03:25.596 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 --format json subsystem list 2024-09-02T12:03:25.682 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:25Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:26.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:26 smithi064 ceph-mon[17163]: pgmap v294: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:26.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-mon[20404]: pgmap v294: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:26.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:26] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:03:26.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: states { 2024-09-02T12:03:26.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: grp_id: 3 2024-09-02T12:03:26.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: state: OPTIMIZED 2024-09-02T12:03:26.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: } 2024-09-02T12:03:26.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: ], , client address: IPv4 172.21.15.94:52306 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 1, 2024-09-02T12:03:26.395 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 2040, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 1, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 2040, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:26.396 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 1, 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 2040, 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:03:26.397 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:03:26.398 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:03:26.398 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:03:26.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:26] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:03:26.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: states { 2024-09-02T12:03:26.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: grp_id: 4 2024-09-02T12:03:26.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: state: OPTIMIZED 2024-09-02T12:03:26.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: } 2024-09-02T12:03:26.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: ], , client address: IPv4 172.21.15.99:37656 2024-09-02T12:03:26.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-mon[21100]: pgmap v294: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-02T12:03:26.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:26] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:03:26.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: states { 2024-09-02T12:03:26.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: grp_id: 2 2024-09-02T12:03:26.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: state: OPTIMIZED 2024-09-02T12:03:26.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: } 2024-09-02T12:03:26.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: ], , client address: IPv4 172.21.15.87:42274 2024-09-02T12:03:26.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:26] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:60952 2024-09-02T12:03:27.232 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:27.232 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.87 2024-09-02T12:03:27.232 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.87 --server-port 5500 --format json subsystem list 2024-09-02T12:03:27.320 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:27Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:27.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:27] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:03:27.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:03:27.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 1 2024-09-02T12:03:27.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:03:27.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:03:27.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:36276 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:28.035 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 2041, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 4080, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 2041, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 4080, 2024-09-02T12:03:28.036 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 2041, 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 4080, 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:28.037 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:03:28.038 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:03:28.038 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:03:28.038 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:03:28.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:28 smithi064 ceph-mon[17163]: pgmap v295: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-02T12:03:28.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:28 smithi094 ceph-mon[20404]: pgmap v295: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-02T12:03:28.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:28 smithi087 ceph-mon[21100]: pgmap v295: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-02T12:03:28.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:28] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:59342 2024-09-02T12:03:28.868 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:28.868 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.94 2024-09-02T12:03:28.868 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.94 --server-port 5500 --format json subsystem list 2024-09-02T12:03:28.954 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:28Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:29.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:29 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:03:29.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:29 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:03:29.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:29 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:03:29.711 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 4081, 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 6120, 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:29.712 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 4081, 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 6120, 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:29.713 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 4081, 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 6120, 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:03:29.714 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:03:29.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:30.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:30.072 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:29 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:29] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:38484 2024-09-02T12:03:30.076 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:30.107 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:30.178 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:30.249 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:30.280 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:30.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:30 smithi094 ceph-mon[20404]: pgmap v296: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-02T12:03:30.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:03:30.344 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:03:30.345 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:03:30.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:30 smithi064 ceph-mon[17163]: pgmap v296: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-02T12:03:30.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:03:30.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:03:30.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:03:30.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:30.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:30.454 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:30.475 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:30 smithi087 ceph-mon[21100]: pgmap v296: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-02T12:03:30.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:03:30.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:03:30.476 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:03:30.485 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:30.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:30.570 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:30.570 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.99 2024-09-02T12:03:30.570 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.99 --server-port 5500 --format json subsystem list 2024-09-02T12:03:30.662 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:30Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:31.435 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:03:31.435 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 6121, 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 8160, 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:31.436 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 6121, 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 8160, 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:03:31.437 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 6121, 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 8160, 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256, 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 0 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:03:31.438 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:03:31.687 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:31 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:31] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:49420 2024-09-02T12:03:32.255 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:32.256 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.64 2024-09-02T12:03:32.256 INFO:tasks.workunit.client.0.smithi038.stderr:+ name=smithi064 2024-09-02T12:03:32.256 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:32.256 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:32.257 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:32.257 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.64 and name smithi064 2024-09-02T12:03:32.257 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.64 and name smithi064' 2024-09-02T12:03:32.257 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name smithi064 --traddr 172.21.15.64 --trsvcid 4420 2024-09-02T12:03:32.332 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:32Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:32.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:32 smithi064 ceph-mon[17163]: pgmap v297: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-02T12:03:32.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:32 smithi094 ceph-mon[20404]: pgmap v297: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-02T12:03:32.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:32 smithi087 ceph-mon[21100]: pgmap v297: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-02T12:03:33.060 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 172.21.15.64:4420: Successful 2024-09-02T12:03:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:33] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.64:4420, context: , client address: IPv4 172.21.15.38:49730 2024-09-02T12:03:33.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:03:33.033188] tcp.c: 951:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 172.21.15.64 port 4420 *** 2024-09-02T12:03:33.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:33 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:33] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.64:4420, context: None 2024-09-02T12:03:33.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:33 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:33] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.64:4420, context: None 2024-09-02T12:03:33.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:33 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:33] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.64:4420, context: None 2024-09-02T12:03:33.934 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:33.934 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.64 and name smithi064 2024-09-02T12:03:33.934 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:33.935 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.64 and name smithi064' 2024-09-02T12:03:33.935 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name smithi064 --traddr 172.21.15.64 --trsvcid 4420 2024-09-02T12:03:34.012 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:34Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:34.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:34 smithi064 ceph-mon[17163]: pgmap v298: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-02T12:03:34.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:34 smithi094 ceph-mon[20404]: pgmap v298: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-02T12:03:34.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:34 smithi087 ceph-mon[21100]: pgmap v298: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-02T12:03:34.764 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 172.21.15.64:4420: Successful 2024-09-02T12:03:35.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:34] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.64:4420, context: , client address: IPv4 172.21.15.38:55070 2024-09-02T12:03:35.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:34] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.64:4420, context: None 2024-09-02T12:03:35.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:34 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:34] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.64:4420, context: None 2024-09-02T12:03:35.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:34] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.64:4420, context: None 2024-09-02T12:03:35.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:35.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:35.603 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:35.603 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:35.604 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.64 and name smithi064 2024-09-02T12:03:35.604 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.64 and name smithi064' 2024-09-02T12:03:35.605 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name smithi064 --traddr 172.21.15.64 --trsvcid 4420 2024-09-02T12:03:35.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:35.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:35.689 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:35Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:35.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:35.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:35.858 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:35.929 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:36.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:36.030 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:36.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:36.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:36.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:36 smithi064 ceph-mon[17163]: pgmap v299: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-02T12:03:36.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:36 smithi094 ceph-mon[20404]: pgmap v299: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-02T12:03:36.439 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:36 smithi087 ceph-mon[21100]: pgmap v299: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-02T12:03:36.440 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 172.21.15.64:4420: Successful 2024-09-02T12:03:36.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:36 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:36] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.64:4420, context: None 2024-09-02T12:03:36.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:36 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:36] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.64:4420, context: None 2024-09-02T12:03:36.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:36 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:36] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.64:4420, context: , client address: IPv4 172.21.15.38:54224 2024-09-02T12:03:36.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:36 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:36] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.64:4420, context: None 2024-09-02T12:03:37.306 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:37.306 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.87 2024-09-02T12:03:37.306 INFO:tasks.workunit.client.0.smithi038.stderr:+ name=smithi087 2024-09-02T12:03:37.306 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:37.307 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:37.307 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:37.307 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.87 and name smithi087 2024-09-02T12:03:37.308 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.87 and name smithi087' 2024-09-02T12:03:37.308 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.87 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name smithi087 --traddr 172.21.15.87 --trsvcid 4420 2024-09-02T12:03:37.396 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:37Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:38.148 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 172.21.15.87:4420: Successful 2024-09-02T12:03:38.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:38 smithi064 ceph-mon[17163]: pgmap v300: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:38.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:38] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.87:4420, context: None 2024-09-02T12:03:38.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:38 smithi094 ceph-mon[20404]: pgmap v300: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:38.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:38 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:38] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.87:4420, context: None 2024-09-02T12:03:38.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:38] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.87:4420, context: None 2024-09-02T12:03:38.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:38 smithi087 ceph-mon[21100]: pgmap v300: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:38.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:38] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.87:4420, context: , client address: IPv4 172.21.15.38:46194 2024-09-02T12:03:38.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:03:38.121113] tcp.c: 951:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 172.21.15.87 port 4420 *** 2024-09-02T12:03:39.018 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:39.018 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.87 and name smithi087 2024-09-02T12:03:39.018 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:39.019 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.87 and name smithi087' 2024-09-02T12:03:39.019 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.87 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name smithi087 --traddr 172.21.15.87 --trsvcid 4420 2024-09-02T12:03:39.096 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:39.836 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 172.21.15.87:4420: Successful 2024-09-02T12:03:40.089 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:39] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.87:4420, context: None 2024-09-02T12:03:40.090 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:39 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:39] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.87:4420, context: , client address: IPv4 172.21.15.38:56786 2024-09-02T12:03:40.090 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:39 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:39] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.87:4420, context: None 2024-09-02T12:03:40.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:39 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:39] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.87:4420, context: None 2024-09-02T12:03:40.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:40 smithi064 ceph-mon[17163]: pgmap v301: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-02T12:03:40.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:40 smithi094 ceph-mon[20404]: pgmap v301: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-02T12:03:40.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:40 smithi087 ceph-mon[21100]: pgmap v301: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-02T12:03:40.661 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:40.661 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.87 and name smithi087 2024-09-02T12:03:40.662 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:40.662 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.87 and name smithi087' 2024-09-02T12:03:40.662 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.87 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name smithi087 --traddr 172.21.15.87 --trsvcid 4420 2024-09-02T12:03:40.749 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:40Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:41.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:41.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:41.224 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:41.256 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:41.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:41.399 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:41.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:41.499 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:41 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:41] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.87:4420, context: , client address: IPv4 172.21.15.38:35578 2024-09-02T12:03:41.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:41.504 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 172.21.15.87:4420: Successful 2024-09-02T12:03:41.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:41.603 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:41.633 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:41.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:41.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:41] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.87:4420, context: None 2024-09-02T12:03:41.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:41 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:41] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.87:4420, context: None 2024-09-02T12:03:41.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:41 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:41] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.87:4420, context: None 2024-09-02T12:03:42.322 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:42.322 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.94 2024-09-02T12:03:42.322 INFO:tasks.workunit.client.0.smithi038.stderr:+ name=smithi094 2024-09-02T12:03:42.322 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:42.323 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:42.323 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:42.324 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.94 and name smithi094 2024-09-02T12:03:42.324 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.94 and name smithi094' 2024-09-02T12:03:42.324 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.94 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name smithi094 --traddr 172.21.15.94 --trsvcid 4420 2024-09-02T12:03:42.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:42 smithi064 ceph-mon[17163]: pgmap v302: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 853 B/s wr, 11 op/s 2024-09-02T12:03:42.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:42 smithi094 ceph-mon[20404]: pgmap v302: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 853 B/s wr, 11 op/s 2024-09-02T12:03:42.393 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:42Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:42.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:42 smithi087 ceph-mon[21100]: pgmap v302: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 853 B/s wr, 11 op/s 2024-09-02T12:03:43.162 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 172.21.15.94:4420: Successful 2024-09-02T12:03:43.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:43 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:43] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.94:4420, context: None 2024-09-02T12:03:43.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:43 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:43] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.94:4420, context: None 2024-09-02T12:03:43.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:43 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:43] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.94:4420, context: None 2024-09-02T12:03:43.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:43 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:43] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.94:4420, context: , client address: IPv4 172.21.15.38:53920 2024-09-02T12:03:43.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:43 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:03:43.134684] tcp.c: 951:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 172.21.15.94 port 4420 *** 2024-09-02T12:03:44.017 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:44.018 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.94 and name smithi094 2024-09-02T12:03:44.018 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:44.018 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.94 and name smithi094' 2024-09-02T12:03:44.018 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.94 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name smithi094 --traddr 172.21.15.94 --trsvcid 4420 2024-09-02T12:03:44.093 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:44Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:44.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:44 smithi087 ceph-mon[21100]: pgmap v303: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:44 smithi064 ceph-mon[17163]: pgmap v303: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:44.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:44 smithi094 ceph-mon[20404]: pgmap v303: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:44.837 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 172.21.15.94:4420: Successful 2024-09-02T12:03:45.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:44] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.94:4420, context: None 2024-09-02T12:03:45.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:44] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.94:4420, context: , client address: IPv4 172.21.15.38:38110 2024-09-02T12:03:45.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:44 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:44] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.94:4420, context: None 2024-09-02T12:03:45.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:44] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.94:4420, context: None 2024-09-02T12:03:45.730 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:45.730 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.94 and name smithi094 2024-09-02T12:03:45.730 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:45.731 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.94 and name smithi094' 2024-09-02T12:03:45.731 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.94 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name smithi094 --traddr 172.21.15.94 --trsvcid 4420 2024-09-02T12:03:45.819 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:45Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:46.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:46 smithi087 ceph-mon[21100]: pgmap v304: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:46.517 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:46 smithi094 ceph-mon[20404]: pgmap v304: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:46.547 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:46 smithi064 ceph-mon[17163]: pgmap v304: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 9.0 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-02T12:03:46.549 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 172.21.15.94:4420: Successful 2024-09-02T12:03:46.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:46.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:46.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:46.817 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:46 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:46] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.94:4420, context: None 2024-09-02T12:03:46.827 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:46.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:46] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.94:4420, context: None 2024-09-02T12:03:46.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:46 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:46] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.94:4420, context: , client address: IPv4 172.21.15.38:44620 2024-09-02T12:03:46.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:46.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:46 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:46] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.94:4420, context: None 2024-09-02T12:03:46.970 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:47.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:47.073 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:47.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:47.176 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:47.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:47.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:47.390 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:03:47.390 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.99 2024-09-02T12:03:47.390 INFO:tasks.workunit.client.0.smithi038.stderr:+ name=smithi099 2024-09-02T12:03:47.391 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:47.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:47.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:47.392 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.99 and name smithi099 2024-09-02T12:03:47.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.99 and name smithi099' 2024-09-02T12:03:47.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.99 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode1 --host-name smithi099 --traddr 172.21.15.99 --trsvcid 4420 2024-09-02T12:03:47.486 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:47Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:48.252 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode1 listener at 172.21.15.99:4420: Successful 2024-09-02T12:03:48.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:48 smithi087 ceph-mon[21100]: pgmap v305: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-09-02T12:03:48.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:48 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:48] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.99:4420, context: None 2024-09-02T12:03:48.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:48 smithi064 ceph-mon[17163]: pgmap v305: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-09-02T12:03:48.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:48 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:48] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.99:4420, context: None 2024-09-02T12:03:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:48 smithi094 ceph-mon[20404]: pgmap v305: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-09-02T12:03:48.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:48 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:48] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.99:4420, context: None 2024-09-02T12:03:48.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:48 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:48] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.99:4420, context: , client address: IPv4 172.21.15.38:50348 2024-09-02T12:03:48.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:48 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:03:48.226038] tcp.c: 951:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 172.21.15.99 port 4420 *** 2024-09-02T12:03:49.158 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:49.158 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:49.159 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.99 and name smithi099 2024-09-02T12:03:49.159 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.99 and name smithi099' 2024-09-02T12:03:49.159 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.99 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode2 --host-name smithi099 --traddr 172.21.15.99 --trsvcid 4420 2024-09-02T12:03:49.245 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:49Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:49.965 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode2 listener at 172.21.15.99:4420: Successful 2024-09-02T12:03:50.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:49 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:49] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.99:4420, context: , client address: IPv4 172.21.15.38:48326 2024-09-02T12:03:50.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:50 smithi087 ceph-mon[21100]: pgmap v306: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:50.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:49] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.99:4420, context: None 2024-09-02T12:03:50.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:50 smithi064 ceph-mon[17163]: pgmap v306: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:50.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:49] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.99:4420, context: None 2024-09-02T12:03:50.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:50 smithi094 ceph-mon[20404]: pgmap v306: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:50.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:49] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.99:4420, context: None 2024-09-02T12:03:50.828 INFO:tasks.workunit.client.0.smithi038.stderr:+ for j in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:50.828 INFO:tasks.workunit.client.0.smithi038.stdout:Adding gateway listener with IP 172.21.15.99 and name smithi099 2024-09-02T12:03:50.828 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:50.828 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Adding gateway listener with IP 172.21.15.99 and name smithi099' 2024-09-02T12:03:50.829 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.99 --server-port 5500 listener add --subsystem nqn.2016-06.io.spdk:cnode3 --host-name smithi099 --traddr 172.21.15.99 --trsvcid 4420 2024-09-02T12:03:50.922 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:50Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:51.705 INFO:tasks.workunit.client.0.smithi038.stdout:Adding nqn.2016-06.io.spdk:cnode3 listener at 172.21.15.99:4420: Successful 2024-09-02T12:03:51.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:51 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:51] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.99:4420, context: , client address: IPv4 172.21.15.38:52246 2024-09-02T12:03:52.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:51] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.99:4420, context: None 2024-09-02T12:03:52.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:51] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.99:4420, context: None 2024-09-02T12:03:52.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:51] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.99:4420, context: None 2024-09-02T12:03:52.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:52.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:52.365 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:52 smithi064 ceph-mon[17163]: pgmap v307: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-09-02T12:03:52.373 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:52.395 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:52 smithi087 ceph-mon[21100]: pgmap v307: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-09-02T12:03:52.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:52.477 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:52.548 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:52.569 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:52 smithi094 ceph-mon[20404]: pgmap v307: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-09-02T12:03:52.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:52.598 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:52.599 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:52.599 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:52.599 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode1 --host '*' 2024-09-02T12:03:52.650 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:52.669 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:52Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:52.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:52.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:52.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:52.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:53.405 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:53 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:53] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 172.21.15.38:46606 2024-09-02T12:03:53.415 INFO:tasks.workunit.client.0.smithi038.stdout:Allowing open host access to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:03:53.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:53 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:53] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2024-09-02T12:03:53.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:53 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:53] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2024-09-02T12:03:53.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:53 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:53] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2024-09-02T12:03:54.234 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:54.234 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:03:54.234 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode2 --host '*' 2024-09-02T12:03:54.323 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:54Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:54.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:54 smithi087 ceph-mon[21100]: pgmap v308: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:54.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:54 smithi064 ceph-mon[17163]: pgmap v308: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:54.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:54 smithi094 ceph-mon[20404]: pgmap v308: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:55.077 INFO:tasks.workunit.client.0.smithi038.stdout:Allowing open host access to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:03:55.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:55 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:55] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 172.21.15.38:41382 2024-09-02T12:03:55.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:55 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:55] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2024-09-02T12:03:55.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:55 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:55] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2024-09-02T12:03:55.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:55 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:55] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2024-09-02T12:03:55.987 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:55.987 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:03:55.988 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode3 --host '*' 2024-09-02T12:03:56.069 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:56Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:56.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:56 smithi087 ceph-mon[21100]: pgmap v309: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:56.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:56 smithi064 ceph-mon[17163]: pgmap v309: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:56.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:56 smithi094 ceph-mon[20404]: pgmap v309: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:03:56.801 INFO:tasks.workunit.client.0.smithi038.stdout:Allowing open host access to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:03:57.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:56] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 172.21.15.38:54896 2024-09-02T12:03:57.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:56] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2024-09-02T12:03:57.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:56 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:56] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2024-09-02T12:03:57.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:56] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2024-09-02T12:03:57.646 INFO:tasks.workunit.client.0.smithi038.stderr:+ image_index=1 2024-09-02T12:03:57.646 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:03:57.647 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:03:57.647 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:03:57.647 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 20 2024-09-02T12:03:57.648 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:03:57.648 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage1 2024-09-02T12:03:57.648 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage1 --load-balancing-group 2 2024-09-02T12:03:57.741 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:57Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:57.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:03:57.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:03:57.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:03:57.977 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:03:58.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:03:58.120 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:03:58.151 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:03:58.222 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:03:58.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-mon[21100]: pgmap v310: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-09-02T12:03:58.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:03:58.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:03:58.353 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:03:58.374 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-mon[20404]: pgmap v310: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-09-02T12:03:58.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:03:58.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-mon[17163]: pgmap v310: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-09-02T12:03:58.570 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 1 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:58] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:58] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e8754-98d1-4474-a559-ac49989ff693 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:03:58.581666] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_2_0, nonce 172.21.15.87:0/1574139620, rc 0 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:03:58.581778] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_2_0, nonce 172.21.15.87:0/1574139620 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:58] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_2_0' nonce='172.21.15.87:0/1574139620' anagrp=2 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 1 2024-09-02T12:03:58.596 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:03:58.595435] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 rbd disk to lun 2024-09-02T12:03:58.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:58] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, context: , client address: IPv4 172.21.15.38:57498 2024-09-02T12:03:58.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:58] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:03:58.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:58] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e8754-98d1-4474-a559-ac49989ff693 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:57498 2024-09-02T12:03:58.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:03:58.541228] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_2_0, nonce 172.21.15.64:0/50897424, rc 0 2024-09-02T12:03:58.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:03:58.541342] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_2_0, nonce 172.21.15.64:0/50897424 2024-09-02T12:03:58.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:58] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_2_0' nonce='172.21.15.64:0/50897424' anagrp=2 2024-09-02T12:03:58.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 1 2024-09-02T12:03:58.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:03:58.553866] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 rbd disk to lun 2024-09-02T12:03:58.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:03:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:03:58] INFO grpc.py:770 (2): Received request to add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2, context: , client address: IPv4 172.21.15.38:57498 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:58] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:58] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e8754-98d1-4474-a559-ac49989ff693 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:03:58.579272] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_2_0, nonce 172.21.15.94:0/1428313099, rc 0 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:03:58.579537] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_2_0, nonce 172.21.15.94:0/1428313099 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:58] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_2_0' nonce='172.21.15.94:0/1428313099' anagrp=2 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 1 2024-09-02T12:03:58.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:03:58.591499] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 rbd disk to lun 2024-09-02T12:03:58.847 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:03:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:03:58] INFO grpc.py:770 (2): Received request to add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:58] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:58] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e8754-98d1-4474-a559-ac49989ff693 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:03:58.582561] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_2_0, nonce 172.21.15.99:0/2825194079, rc 0 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:03:58.582638] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_2_0, nonce 172.21.15.99:0/2825194079 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:58] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_2_0' nonce='172.21.15.99:0/2825194079' anagrp=2 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 1 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:03:58.597975] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 rbd disk to lun 2024-09-02T12:03:58.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:03:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:03:58] INFO grpc.py:770 (2): Received request to add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:03:59.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:03:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:03:58] INFO grpc.py:770 (2): Received request to add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:03:59.399 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:03:59.400 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:03:59.400 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage2 2024-09-02T12:03:59.400 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage2 --load-balancing-group 3 2024-09-02T12:03:59.495 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:03:59Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:03:59.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:03:59 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2873961118' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:03:59.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:03:59 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2873961118' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:03:59.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:03:59 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2873961118' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:00.287 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 2 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:00.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-mon[21100]: pgmap v311: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:04:00.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:00] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:00.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:00] INFO grpc.py:275 (2): Received request to create bdev bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:00.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:00.295353] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_3_0, nonce 172.21.15.87:0/2093777871, rc 0 2024-09-02T12:04:00.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:00.295802] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_3_0, nonce 172.21.15.87:0/2093777871 2024-09-02T12:04:00.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:00] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_3_0' nonce='172.21.15.87:0/2093777871' anagrp=3 2024-09-02T12:04:00.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:00] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 1 2024-09-02T12:04:00.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:00.307550] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 rbd disk to lun 2024-09-02T12:04:00.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:00 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:00] INFO grpc.py:770 (2): Received request to add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:04:00.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-mon[17163]: pgmap v311: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:04:00.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:00] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, context: , client address: IPv4 172.21.15.38:52276 2024-09-02T12:04:00.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:00] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:00.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:00] INFO grpc.py:275 (2): Received request to create bdev bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:52276 2024-09-02T12:04:00.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:00.257590] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_3_0, nonce 172.21.15.64:0/1780222602, rc 0 2024-09-02T12:04:00.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:00.257791] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_3_0, nonce 172.21.15.64:0/1780222602 2024-09-02T12:04:00.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:00] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_3_0' nonce='172.21.15.64:0/1780222602' anagrp=3 2024-09-02T12:04:00.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:00] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 1 2024-09-02T12:04:00.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:00.269822] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 rbd disk to lun 2024-09-02T12:04:00.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:00 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:00] INFO grpc.py:770 (2): Received request to add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3, context: , client address: IPv4 172.21.15.38:52276 2024-09-02T12:04:00.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-mon[20404]: pgmap v311: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:00] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:00] INFO grpc.py:275 (2): Received request to create bdev bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:00.295096] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_3_0, nonce 172.21.15.94:0/3575641970, rc 0 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:00.295201] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_3_0, nonce 172.21.15.94:0/3575641970 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:00] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_3_0' nonce='172.21.15.94:0/3575641970' anagrp=3 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:00] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 1 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:00.308525] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 rbd disk to lun 2024-09-02T12:04:00.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:00] INFO grpc.py:770 (2): Received request to add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:00] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:00] INFO grpc.py:275 (2): Received request to create bdev bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:00.297739] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_3_0, nonce 172.21.15.99:0/1484662819, rc 0 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:00.297811] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_3_0, nonce 172.21.15.99:0/1484662819 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:00] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_3_0' nonce='172.21.15.99:0/1484662819' anagrp=3 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:00] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 1 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:00.312132] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 rbd disk to lun 2024-09-02T12:04:00.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:00 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:00] INFO grpc.py:770 (2): Received request to add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:04:01.135 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:01.135 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:01.135 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage3 2024-09-02T12:04:01.135 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage3 --load-balancing-group 4 2024-09-02T12:04:01.210 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:01Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:02.008 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 3 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:02 smithi064 ceph-mon[17163]: pgmap v312: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 1.5 KiB/s wr, 19 op/s 2024-09-02T12:04:02.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:01] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, context: , client address: IPv4 172.21.15.38:39304 2024-09-02T12:04:02.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:01] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:01] INFO grpc.py:275 (2): Received request to create bdev bdev_f96044ec-462e-4f71-a581-9eab30b51024 from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:39304 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:01.981645] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_4_0, nonce 172.21.15.64:0/1969798782, rc 0 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:01.981739] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_4_0, nonce 172.21.15.64:0/1969798782 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:01] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_4_0' nonce='172.21.15.64:0/1969798782' anagrp=4 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:01] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 1 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:01.993845] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_f96044ec-462e-4f71-a581-9eab30b51024 rbd disk to lun 2024-09-02T12:04:02.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:01] INFO grpc.py:770 (2): Received request to add bdev_f96044ec-462e-4f71-a581-9eab30b51024 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4, context: , client address: IPv4 172.21.15.38:39304 2024-09-02T12:04:02.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-mon[20404]: pgmap v312: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 1.5 KiB/s wr, 19 op/s 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:02] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:02] INFO grpc.py:275 (2): Received request to create bdev bdev_f96044ec-462e-4f71-a581-9eab30b51024 from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:02.028201] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_4_0, nonce 172.21.15.94:0/4173031011, rc 0 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:02.028289] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_4_0, nonce 172.21.15.94:0/4173031011 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_4_0' nonce='172.21.15.94:0/4173031011' anagrp=4 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 1 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:02.041390] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_f96044ec-462e-4f71-a581-9eab30b51024 rbd disk to lun 2024-09-02T12:04:02.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:02] INFO grpc.py:770 (2): Received request to add bdev_f96044ec-462e-4f71-a581-9eab30b51024 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:02] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:02] INFO grpc.py:275 (2): Received request to create bdev bdev_f96044ec-462e-4f71-a581-9eab30b51024 from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:02.030189] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_4_0, nonce 172.21.15.99:0/1564998870, rc 0 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:02.030260] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_4_0, nonce 172.21.15.99:0/1564998870 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_4_0' nonce='172.21.15.99:0/1564998870' anagrp=4 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 1 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:02.044562] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_f96044ec-462e-4f71-a581-9eab30b51024 rbd disk to lun 2024-09-02T12:04:02.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:02 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:02] INFO grpc.py:770 (2): Received request to add bdev_f96044ec-462e-4f71-a581-9eab30b51024 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:04:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-mon[21100]: pgmap v312: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 1.5 KiB/s wr, 19 op/s 2024-09-02T12:04:02.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:02] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:02.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:02] INFO grpc.py:275 (2): Received request to create bdev bdev_f96044ec-462e-4f71-a581-9eab30b51024 from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:02.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:02.027389] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_4_0, nonce 172.21.15.87:0/2628170397, rc 0 2024-09-02T12:04:02.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:02.027673] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_4_0, nonce 172.21.15.87:0/2628170397 2024-09-02T12:04:02.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_4_0' nonce='172.21.15.87:0/2628170397' anagrp=4 2024-09-02T12:04:02.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 1 2024-09-02T12:04:02.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:02.040208] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_f96044ec-462e-4f71-a581-9eab30b51024 rbd disk to lun 2024-09-02T12:04:02.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:02 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:02] INFO grpc.py:770 (2): Received request to add bdev_f96044ec-462e-4f71-a581-9eab30b51024 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:04:02.838 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:02.838 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:02.838 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage4 2024-09-02T12:04:02.839 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage4 --load-balancing-group 1 2024-09-02T12:04:02.931 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:02Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:03.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:03.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:03.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:03.548 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:03.620 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:03.645 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 4 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:03.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:03.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:03.795 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:03.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:03] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, context: , client address: IPv4 172.21.15.38:51228 2024-09-02T12:04:03.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:03] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:03.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:03] INFO grpc.py:275 (2): Received request to create bdev bdev_036643fa-ace3-4623-acfe-a405d326e7c7 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:51228 2024-09-02T12:04:03.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:03.617360] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_1_0, nonce 172.21.15.64:0/1613028247, rc 0 2024-09-02T12:04:03.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:03.617463] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_1_0, nonce 172.21.15.64:0/1613028247 2024-09-02T12:04:03.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:03] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_1_0' nonce='172.21.15.64:0/1613028247' anagrp=1 2024-09-02T12:04:03.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 1 2024-09-02T12:04:03.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:03.630487] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 rbd disk to lun 2024-09-02T12:04:03.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:03] INFO grpc.py:770 (2): Received request to add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1, context: , client address: IPv4 172.21.15.38:51228 2024-09-02T12:04:03.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:03.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:03.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:03] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:03.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:03] INFO grpc.py:275 (2): Received request to create bdev bdev_036643fa-ace3-4623-acfe-a405d326e7c7 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:03.919 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:03.655597] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_1_0, nonce 172.21.15.87:0/21871447, rc 0 2024-09-02T12:04:03.919 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:03.655687] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_1_0, nonce 172.21.15.87:0/21871447 2024-09-02T12:04:03.919 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:03] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_1_0' nonce='172.21.15.87:0/21871447' anagrp=1 2024-09-02T12:04:03.919 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 1 2024-09-02T12:04:03.919 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:03.669536] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 rbd disk to lun 2024-09-02T12:04:03.919 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:03] INFO grpc.py:770 (2): Received request to add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:04:03.928 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:03] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:03] INFO grpc.py:275 (2): Received request to create bdev bdev_036643fa-ace3-4623-acfe-a405d326e7c7 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:03.652672] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_1_0, nonce 172.21.15.94:0/2486214710, rc 0 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:03.652770] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_1_0, nonce 172.21.15.94:0/2486214710 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:03] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_1_0' nonce='172.21.15.94:0/2486214710' anagrp=1 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 1 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:03.666357] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 rbd disk to lun 2024-09-02T12:04:03.949 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:03] INFO grpc.py:770 (2): Received request to add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:04:03.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:03.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:03] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:03.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:03] INFO grpc.py:275 (2): Received request to create bdev bdev_036643fa-ace3-4623-acfe-a405d326e7c7 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:03.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:03.655942] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_1_0, nonce 172.21.15.99:0/2392623721, rc 0 2024-09-02T12:04:03.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:03.656011] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_1_0, nonce 172.21.15.99:0/2392623721 2024-09-02T12:04:03.960 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:03] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_1_0' nonce='172.21.15.99:0/2392623721' anagrp=1 2024-09-02T12:04:03.960 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 1 2024-09-02T12:04:03.960 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:03.673186] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 rbd disk to lun 2024-09-02T12:04:03.960 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:03] INFO grpc.py:770 (2): Received request to add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:04:04.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:04 smithi087 ceph-mon[21100]: pgmap v313: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1023 B/s wr, 14 op/s 2024-09-02T12:04:04.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:04 smithi094 ceph-mon[20404]: pgmap v313: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1023 B/s wr, 14 op/s 2024-09-02T12:04:04.499 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:04.499 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:04.499 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage5 2024-09-02T12:04:04.499 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage5 --load-balancing-group 2 2024-09-02T12:04:04.589 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:04Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:04.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:04 smithi064 ceph-mon[17163]: pgmap v313: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 1023 B/s wr, 14 op/s 2024-09-02T12:04:05.323 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 5 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:05.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:05 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:05] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, context: , client address: IPv4 172.21.15.38:55482 2024-09-02T12:04:05.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:05 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:05] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:05.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:05 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:05] INFO grpc.py:275 (2): Received request to create bdev bdev_127a264a-a286-4c07-95dc-3c381465fe2e from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:55482 2024-09-02T12:04:05.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:05 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:05] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 2 2024-09-02T12:04:05.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:05 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:05.308074] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_127a264a-a286-4c07-95dc-3c381465fe2e rbd disk to lun 2024-09-02T12:04:05.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:05 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:05] INFO grpc.py:770 (2): Received request to add bdev_127a264a-a286-4c07-95dc-3c381465fe2e to nqn.2016-06.io.spdk:cnode1 with ANA group id 2, context: , client address: IPv4 172.21.15.38:55482 2024-09-02T12:04:05.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:05] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:05.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:05] INFO grpc.py:275 (2): Received request to create bdev bdev_127a264a-a286-4c07-95dc-3c381465fe2e from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:05.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:05] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 2 2024-09-02T12:04:05.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:05.335198] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_127a264a-a286-4c07-95dc-3c381465fe2e rbd disk to lun 2024-09-02T12:04:05.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:05] INFO grpc.py:770 (2): Received request to add bdev_127a264a-a286-4c07-95dc-3c381465fe2e to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:04:05.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:05 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:05] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:05.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:05 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:05] INFO grpc.py:275 (2): Received request to create bdev bdev_127a264a-a286-4c07-95dc-3c381465fe2e from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:05.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:05 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:05] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 2 2024-09-02T12:04:05.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:05 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:05.354768] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_127a264a-a286-4c07-95dc-3c381465fe2e rbd disk to lun 2024-09-02T12:04:05.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:05 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:05] INFO grpc.py:770 (2): Received request to add bdev_127a264a-a286-4c07-95dc-3c381465fe2e to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:04:05.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:05 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:05] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:05.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:05 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:05] INFO grpc.py:275 (2): Received request to create bdev bdev_127a264a-a286-4c07-95dc-3c381465fe2e from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:05.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:05 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:05] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 2 2024-09-02T12:04:05.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:05 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:05.334386] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_127a264a-a286-4c07-95dc-3c381465fe2e rbd disk to lun 2024-09-02T12:04:05.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:05 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:05] INFO grpc.py:770 (2): Received request to add bdev_127a264a-a286-4c07-95dc-3c381465fe2e to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:04:06.127 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:06.127 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:06.127 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage6 2024-09-02T12:04:06.127 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage6 --load-balancing-group 3 2024-09-02T12:04:06.219 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:06Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:06.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:06 smithi087 ceph-mon[21100]: pgmap v314: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-09-02T12:04:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-mon[17163]: pgmap v314: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-09-02T12:04:06.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:06 smithi094 ceph-mon[20404]: pgmap v314: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-09-02T12:04:06.963 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 6 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:07.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:06] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:07.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:06] INFO grpc.py:275 (2): Received request to create bdev bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:07.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 2 2024-09-02T12:04:07.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:06.974960] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 rbd disk to lun 2024-09-02T12:04:07.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:06] INFO grpc.py:770 (2): Received request to add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:04:07.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:06] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, context: , client address: IPv4 172.21.15.38:54382 2024-09-02T12:04:07.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:06] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:07.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:06] INFO grpc.py:275 (2): Received request to create bdev bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:54382 2024-09-02T12:04:07.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 2 2024-09-02T12:04:07.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:06.947974] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 rbd disk to lun 2024-09-02T12:04:07.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:06] INFO grpc.py:770 (2): Received request to add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3, context: , client address: IPv4 172.21.15.38:54382 2024-09-02T12:04:07.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:06] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:07.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:06] INFO grpc.py:275 (2): Received request to create bdev bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:07.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 2 2024-09-02T12:04:07.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:06.972328] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 rbd disk to lun 2024-09-02T12:04:07.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:06] INFO grpc.py:770 (2): Received request to add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:04:07.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:06] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:07.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:06] INFO grpc.py:275 (2): Received request to create bdev bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:07.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 2 2024-09-02T12:04:07.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:06.976275] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 rbd disk to lun 2024-09-02T12:04:07.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:06] INFO grpc.py:770 (2): Received request to add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:04:07.829 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:07.829 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:07.829 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage7 2024-09-02T12:04:07.829 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage7 --load-balancing-group 4 2024-09-02T12:04:07.906 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:07Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:08.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:08 smithi087 ceph-mon[21100]: pgmap v315: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 1.8 KiB/s wr, 55 op/s 2024-09-02T12:04:08.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-mon[17163]: pgmap v315: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 1.8 KiB/s wr, 55 op/s 2024-09-02T12:04:08.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:08 smithi094 ceph-mon[20404]: pgmap v315: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 1.8 KiB/s wr, 55 op/s 2024-09-02T12:04:08.688 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 7 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:08.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:08] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:08.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:08] INFO grpc.py:275 (2): Received request to create bdev bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:08.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 2 2024-09-02T12:04:08.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:08.700031] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b rbd disk to lun 2024-09-02T12:04:08.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:08] INFO grpc.py:770 (2): Received request to add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:04:08.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:09.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:08] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:09.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:08] INFO grpc.py:275 (2): Received request to create bdev bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:09.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 2 2024-09-02T12:04:09.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:08.702801] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b rbd disk to lun 2024-09-02T12:04:09.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:08] INFO grpc.py:770 (2): Received request to add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:04:09.012 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:08] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, context: , client address: IPv4 172.21.15.38:42070 2024-09-02T12:04:09.012 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:08] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:04:09.012 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:08] INFO grpc.py:275 (2): Received request to create bdev bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:42070 2024-09-02T12:04:09.012 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 2 2024-09-02T12:04:09.012 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:08.671577] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b rbd disk to lun 2024-09-02T12:04:09.012 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:08] INFO grpc.py:770 (2): Received request to add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b to nqn.2016-06.io.spdk:cnode1 with ANA group id 4, context: , client address: IPv4 172.21.15.38:42070 2024-09-02T12:04:09.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:09.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:09.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:08] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:09.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:08] INFO grpc.py:275 (2): Received request to create bdev bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:09.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 2 2024-09-02T12:04:09.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:08.700844] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b rbd disk to lun 2024-09-02T12:04:09.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:08] INFO grpc.py:770 (2): Received request to add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:04:09.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:09.196 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:09.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:09.298 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:09.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:09 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3216781218' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:09.370 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:09.431 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:09 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3216781218' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:09.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:09.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:09.496 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:09 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3216781218' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:09.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:09.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:09.540 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:09.540 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:09.540 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage8 2024-09-02T12:04:09.540 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage8 --load-balancing-group 1 2024-09-02T12:04:09.622 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:09Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:10.395 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 8 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:10.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:10 smithi087 ceph-mon[21100]: pgmap v316: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 1.4 KiB/s wr, 59 op/s 2024-09-02T12:04:10.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:10 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:10] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:10.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:10 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:10] INFO grpc.py:275 (2): Received request to create bdev bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:10.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:10 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:10] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 2 2024-09-02T12:04:10.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:10 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:10.409597] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 rbd disk to lun 2024-09-02T12:04:10.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:10 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:10] INFO grpc.py:770 (2): Received request to add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:04:10.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-mon[17163]: pgmap v316: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 1.4 KiB/s wr, 59 op/s 2024-09-02T12:04:10.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:10] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, context: , client address: IPv4 172.21.15.38:39740 2024-09-02T12:04:10.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:10] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:10.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:10] INFO grpc.py:275 (2): Received request to create bdev bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:39740 2024-09-02T12:04:10.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:10] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 2 2024-09-02T12:04:10.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:10.378327] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 rbd disk to lun 2024-09-02T12:04:10.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:10 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:10] INFO grpc.py:770 (2): Received request to add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1, context: , client address: IPv4 172.21.15.38:39740 2024-09-02T12:04:10.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:10 smithi094 ceph-mon[20404]: pgmap v316: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 1.4 KiB/s wr, 59 op/s 2024-09-02T12:04:10.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:10] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:10.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:10] INFO grpc.py:275 (2): Received request to create bdev bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:10.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:10] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 2 2024-09-02T12:04:10.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:10.407127] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 rbd disk to lun 2024-09-02T12:04:10.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:10] INFO grpc.py:770 (2): Received request to add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:04:10.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:10 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:10] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:10.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:10 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:10] INFO grpc.py:275 (2): Received request to create bdev bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:10.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:10 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:10] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 2 2024-09-02T12:04:10.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:10 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:10.406296] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 rbd disk to lun 2024-09-02T12:04:10.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:10 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:10] INFO grpc.py:770 (2): Received request to add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:04:11.324 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:11.325 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:11.325 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage9 2024-09-02T12:04:11.325 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage9 --load-balancing-group 2 2024-09-02T12:04:11.414 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:11Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:12.146 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 9 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:12.396 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-mon[17163]: pgmap v317: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 2.0 KiB/s wr, 74 op/s 2024-09-02T12:04:12.396 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:12] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, context: , client address: IPv4 172.21.15.38:43974 2024-09-02T12:04:12.396 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:12] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:12.397 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:12] INFO grpc.py:275 (2): Received request to create bdev bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:43974 2024-09-02T12:04:12.397 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:12] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 3 2024-09-02T12:04:12.397 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:12.129746] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 rbd disk to lun 2024-09-02T12:04:12.397 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:12 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:12] INFO grpc.py:770 (2): Received request to add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2, context: , client address: IPv4 172.21.15.38:43974 2024-09-02T12:04:12.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:12 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:12] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:12.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:12 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:12] INFO grpc.py:275 (2): Received request to create bdev bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:12.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:12 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:12] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 3 2024-09-02T12:04:12.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:12 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:12.154707] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 rbd disk to lun 2024-09-02T12:04:12.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:12 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:12] INFO grpc.py:770 (2): Received request to add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:04:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:12 smithi087 ceph-mon[21100]: pgmap v317: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 2.0 KiB/s wr, 74 op/s 2024-09-02T12:04:12.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:12 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:12] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:12.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:12 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:12] INFO grpc.py:275 (2): Received request to create bdev bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:12.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:12 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:12] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 3 2024-09-02T12:04:12.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:12 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:12.159082] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 rbd disk to lun 2024-09-02T12:04:12.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:12 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:12] INFO grpc.py:770 (2): Received request to add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:04:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:12 smithi094 ceph-mon[20404]: pgmap v317: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 2.0 KiB/s wr, 74 op/s 2024-09-02T12:04:12.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:12] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:12.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:12] INFO grpc.py:275 (2): Received request to create bdev bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:12.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:12] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 3 2024-09-02T12:04:12.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:12.155938] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 rbd disk to lun 2024-09-02T12:04:12.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:12] INFO grpc.py:770 (2): Received request to add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:04:12.977 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:12.977 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:12.977 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage10 2024-09-02T12:04:12.977 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage10 --load-balancing-group 3 2024-09-02T12:04:13.051 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:13Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:13.838 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 10 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:14.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:13] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, context: , client address: IPv4 172.21.15.38:43124 2024-09-02T12:04:14.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:13] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:14.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:13] INFO grpc.py:275 (2): Received request to create bdev bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:43124 2024-09-02T12:04:14.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 3 2024-09-02T12:04:14.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:13.821511] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 rbd disk to lun 2024-09-02T12:04:14.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:13] INFO grpc.py:770 (2): Received request to add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3, context: , client address: IPv4 172.21.15.38:43124 2024-09-02T12:04:14.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:13] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:14.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:13] INFO grpc.py:275 (2): Received request to create bdev bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:14.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 3 2024-09-02T12:04:14.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:13.850002] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 rbd disk to lun 2024-09-02T12:04:14.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:13] INFO grpc.py:770 (2): Received request to add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:04:14.193 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:13] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:14.194 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:13] INFO grpc.py:275 (2): Received request to create bdev bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:14.194 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 3 2024-09-02T12:04:14.194 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:13.849093] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 rbd disk to lun 2024-09-02T12:04:14.194 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:13] INFO grpc.py:770 (2): Received request to add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:04:14.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:13] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:14.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:13] INFO grpc.py:275 (2): Received request to create bdev bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:14.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 3 2024-09-02T12:04:14.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:13.848107] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 rbd disk to lun 2024-09-02T12:04:14.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:13] INFO grpc.py:770 (2): Received request to add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:04:14.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:14 smithi087 ceph-mon[21100]: pgmap v318: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:04:14.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:14.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:14 smithi064 ceph-mon[17163]: pgmap v318: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:04:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:14 smithi094 ceph-mon[20404]: pgmap v318: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:04:14.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:14.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:14.701 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:14.712 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:14.713 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:14.713 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage11 2024-09-02T12:04:14.713 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage11 --load-balancing-group 4 2024-09-02T12:04:14.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:14.818 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:14.843 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:14.874 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:14.945 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:15.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:15.045 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:15.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:15.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:15.569 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 11 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:15.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:15] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, context: , client address: IPv4 172.21.15.38:43192 2024-09-02T12:04:15.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:15] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:15.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:15] INFO grpc.py:275 (2): Received request to create bdev bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:43192 2024-09-02T12:04:15.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:15] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 3 2024-09-02T12:04:15.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:15.550968] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc rbd disk to lun 2024-09-02T12:04:15.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:15] INFO grpc.py:770 (2): Received request to add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc to nqn.2016-06.io.spdk:cnode1 with ANA group id 4, context: , client address: IPv4 172.21.15.38:43192 2024-09-02T12:04:15.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:15] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:15.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:15] INFO grpc.py:275 (2): Received request to create bdev bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:15.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:15] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 3 2024-09-02T12:04:15.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:15.578398] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc rbd disk to lun 2024-09-02T12:04:15.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:15] INFO grpc.py:770 (2): Received request to add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:04:15.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:15 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:15] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:15.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:15 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:15] INFO grpc.py:275 (2): Received request to create bdev bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:15.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:15 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:15] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 3 2024-09-02T12:04:15.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:15 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:15.576690] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc rbd disk to lun 2024-09-02T12:04:15.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:15 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:15] INFO grpc.py:770 (2): Received request to add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:04:16.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:15 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:15] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:16.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:15 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:15] INFO grpc.py:275 (2): Received request to create bdev bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:16.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:15 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:15] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 3 2024-09-02T12:04:16.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:15 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:15.585429] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc rbd disk to lun 2024-09-02T12:04:16.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:15 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:15] INFO grpc.py:770 (2): Received request to add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:04:16.416 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:16.416 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:16.416 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage12 2024-09-02T12:04:16.417 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage12 --load-balancing-group 1 2024-09-02T12:04:16.496 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:16Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:16.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:16 smithi087 ceph-mon[21100]: pgmap v319: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:04:16.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:16 smithi064 ceph-mon[17163]: pgmap v319: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:04:16.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:16 smithi094 ceph-mon[20404]: pgmap v319: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:04:17.237 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 12 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:17.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:17 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:17] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:17.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:17 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:17] INFO grpc.py:275 (2): Received request to create bdev bdev_ba54d78c-a118-4885-be91-cc8c4c33830f from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:17.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:17 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:17] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 3 2024-09-02T12:04:17.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:17 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:17.246916] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f rbd disk to lun 2024-09-02T12:04:17.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:17 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:17] INFO grpc.py:770 (2): Received request to add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:04:17.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:17 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:17] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, context: , client address: IPv4 172.21.15.38:53236 2024-09-02T12:04:17.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:17 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:17] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:17.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:17 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:17] INFO grpc.py:275 (2): Received request to create bdev bdev_ba54d78c-a118-4885-be91-cc8c4c33830f from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:53236 2024-09-02T12:04:17.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:17 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:17] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 3 2024-09-02T12:04:17.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:17 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:17.222288] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f rbd disk to lun 2024-09-02T12:04:17.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:17 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:17] INFO grpc.py:770 (2): Received request to add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f to nqn.2016-06.io.spdk:cnode1 with ANA group id 1, context: , client address: IPv4 172.21.15.38:53236 2024-09-02T12:04:17.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:17] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:17.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:17] INFO grpc.py:275 (2): Received request to create bdev bdev_ba54d78c-a118-4885-be91-cc8c4c33830f from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:17.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:17] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 3 2024-09-02T12:04:17.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:17.247803] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f rbd disk to lun 2024-09-02T12:04:17.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:17] INFO grpc.py:770 (2): Received request to add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:04:17.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:17 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:17] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:17.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:17 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:17] INFO grpc.py:275 (2): Received request to create bdev bdev_ba54d78c-a118-4885-be91-cc8c4c33830f from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:17.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:17 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:17] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 3 2024-09-02T12:04:17.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:17 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:17.251673] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f rbd disk to lun 2024-09-02T12:04:17.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:17 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:17] INFO grpc.py:770 (2): Received request to add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:04:18.034 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:18.034 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:18.034 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage13 2024-09-02T12:04:18.034 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage13 --load-balancing-group 2 2024-09-02T12:04:18.109 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:18Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:18.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:18 smithi087 ceph-mon[21100]: pgmap v320: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 2.2 KiB/s wr, 92 op/s 2024-09-02T12:04:18.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-mon[17163]: pgmap v320: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 2.2 KiB/s wr, 92 op/s 2024-09-02T12:04:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:18 smithi094 ceph-mon[20404]: pgmap v320: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 2.2 KiB/s wr, 92 op/s 2024-09-02T12:04:18.863 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 13 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:19.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:18] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, context: , client address: IPv4 172.21.15.38:35516 2024-09-02T12:04:19.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:18] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:04:19.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:18] INFO grpc.py:275 (2): Received request to create bdev bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:35516 2024-09-02T12:04:19.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 4 2024-09-02T12:04:19.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:18.845332] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 rbd disk to lun 2024-09-02T12:04:19.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:18] INFO grpc.py:770 (2): Received request to add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2, context: , client address: IPv4 172.21.15.38:35516 2024-09-02T12:04:19.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:18] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:19.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:18] INFO grpc.py:275 (2): Received request to create bdev bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:19.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 4 2024-09-02T12:04:19.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:18.875034] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 rbd disk to lun 2024-09-02T12:04:19.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:18] INFO grpc.py:770 (2): Received request to add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:04:19.215 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:18] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:19.215 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:18] INFO grpc.py:275 (2): Received request to create bdev bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:19.215 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 4 2024-09-02T12:04:19.215 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:18.873492] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 rbd disk to lun 2024-09-02T12:04:19.215 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:18] INFO grpc.py:770 (2): Received request to add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:04:19.216 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:18] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:19.216 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:18] INFO grpc.py:275 (2): Received request to create bdev bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:19.216 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 4 2024-09-02T12:04:19.216 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:18.870591] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 rbd disk to lun 2024-09-02T12:04:19.216 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:18] INFO grpc.py:770 (2): Received request to add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:04:19.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:19 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2392550604' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:19.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:19 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2392550604' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:19.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:19 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2392550604' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:19.763 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:19.763 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:19.763 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage14 2024-09-02T12:04:19.763 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage14 --load-balancing-group 3 2024-09-02T12:04:19.849 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:20.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:20.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:20.241 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:20.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:20.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:20.414 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:20.446 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:20.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:20 smithi087 ceph-mon[21100]: pgmap v321: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:04:20.507 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:20 smithi094 ceph-mon[20404]: pgmap v321: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:04:20.517 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:20.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:20.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-mon[17163]: pgmap v321: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:04:20.619 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 14 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:20.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:20.653 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:20.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:20.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:20] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, context: , client address: IPv4 172.21.15.38:57608 2024-09-02T12:04:20.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:20] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:20.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:20] INFO grpc.py:275 (2): Received request to create bdev bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:57608 2024-09-02T12:04:20.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 4 2024-09-02T12:04:20.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:20.601522] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 rbd disk to lun 2024-09-02T12:04:20.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:20] INFO grpc.py:770 (2): Received request to add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3, context: , client address: IPv4 172.21.15.38:57608 2024-09-02T12:04:20.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:20] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:20.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:20] INFO grpc.py:275 (2): Received request to create bdev bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:20.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 4 2024-09-02T12:04:20.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:20.631095] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 rbd disk to lun 2024-09-02T12:04:20.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:20] INFO grpc.py:770 (2): Received request to add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:04:20.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:20] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:20.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:20] INFO grpc.py:275 (2): Received request to create bdev bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:20.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 4 2024-09-02T12:04:20.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:20.627463] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 rbd disk to lun 2024-09-02T12:04:20.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:20] INFO grpc.py:770 (2): Received request to add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:04:21.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:20] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:21.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:20] INFO grpc.py:275 (2): Received request to create bdev bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:21.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 4 2024-09-02T12:04:21.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:20.632212] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 rbd disk to lun 2024-09-02T12:04:21.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:20] INFO grpc.py:770 (2): Received request to add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:04:21.465 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:21.466 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:21.466 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage15 2024-09-02T12:04:21.466 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage15 --load-balancing-group 4 2024-09-02T12:04:21.544 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:21Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:22.308 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 15 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:22.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:22 smithi087 ceph-mon[21100]: pgmap v322: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 2.2 KiB/s wr, 94 op/s 2024-09-02T12:04:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:22] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:22] INFO grpc.py:275 (2): Received request to create bdev bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:22] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 4 2024-09-02T12:04:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:22.316568] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 rbd disk to lun 2024-09-02T12:04:22.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:22 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:22] INFO grpc.py:770 (2): Received request to add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:04:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-mon[17163]: pgmap v322: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 2.2 KiB/s wr, 94 op/s 2024-09-02T12:04:22.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:22] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, context: , client address: IPv4 172.21.15.38:33430 2024-09-02T12:04:22.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:22] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:22.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:22] INFO grpc.py:275 (2): Received request to create bdev bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:33430 2024-09-02T12:04:22.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:22] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 4 2024-09-02T12:04:22.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:22.290799] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 rbd disk to lun 2024-09-02T12:04:22.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:22 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:22] INFO grpc.py:770 (2): Received request to add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4, context: , client address: IPv4 172.21.15.38:33430 2024-09-02T12:04:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:22 smithi094 ceph-mon[20404]: pgmap v322: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 2.2 KiB/s wr, 94 op/s 2024-09-02T12:04:22.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:22] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:22.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:22] INFO grpc.py:275 (2): Received request to create bdev bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:22.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:22] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 4 2024-09-02T12:04:22.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:22.315628] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 rbd disk to lun 2024-09-02T12:04:22.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:22 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:22] INFO grpc.py:770 (2): Received request to add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:04:22.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:22] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:22.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:22] INFO grpc.py:275 (2): Received request to create bdev bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:22.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:22] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 4 2024-09-02T12:04:22.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:22.317654] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 rbd disk to lun 2024-09-02T12:04:22.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:22 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:22] INFO grpc.py:770 (2): Received request to add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:04:23.126 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:23.126 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:23.126 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage16 2024-09-02T12:04:23.126 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage16 --load-balancing-group 1 2024-09-02T12:04:23.199 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:23Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:23.987 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 16 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:24.239 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:23] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, context: , client address: IPv4 172.21.15.38:38622 2024-09-02T12:04:24.239 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:23] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:24.239 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:23] INFO grpc.py:275 (2): Received request to create bdev bdev_1ec594ff-1157-4225-98ec-c6c41b32344b from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:38622 2024-09-02T12:04:24.239 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 4 2024-09-02T12:04:24.240 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:23.971232] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b rbd disk to lun 2024-09-02T12:04:24.240 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:23] INFO grpc.py:770 (2): Received request to add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b to nqn.2016-06.io.spdk:cnode1 with ANA group id 1, context: , client address: IPv4 172.21.15.38:38622 2024-09-02T12:04:24.240 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:23] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:24.240 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:23] INFO grpc.py:275 (2): Received request to create bdev bdev_1ec594ff-1157-4225-98ec-c6c41b32344b from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:24.240 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 4 2024-09-02T12:04:24.240 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:23.998516] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b rbd disk to lun 2024-09-02T12:04:24.241 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:23] INFO grpc.py:770 (2): Received request to add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:04:24.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:23] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:24.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:23] INFO grpc.py:275 (2): Received request to create bdev bdev_1ec594ff-1157-4225-98ec-c6c41b32344b from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:24.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 4 2024-09-02T12:04:24.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:23.996402] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b rbd disk to lun 2024-09-02T12:04:24.241 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:23] INFO grpc.py:770 (2): Received request to add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:04:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:23] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:23] INFO grpc.py:275 (2): Received request to create bdev bdev_1ec594ff-1157-4225-98ec-c6c41b32344b from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 4 2024-09-02T12:04:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:24.000756] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b rbd disk to lun 2024-09-02T12:04:24.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:24 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:24] INFO grpc.py:770 (2): Received request to add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:04:24.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:24 smithi087 ceph-mon[21100]: pgmap v323: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 1.7 KiB/s wr, 88 op/s 2024-09-02T12:04:24.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:24 smithi064 ceph-mon[17163]: pgmap v323: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 1.7 KiB/s wr, 88 op/s 2024-09-02T12:04:24.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:24 smithi094 ceph-mon[20404]: pgmap v323: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 1.7 KiB/s wr, 88 op/s 2024-09-02T12:04:24.848 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:24.848 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:24.848 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage17 2024-09-02T12:04:24.848 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage17 --load-balancing-group 2 2024-09-02T12:04:24.920 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:24Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:25.663 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 17 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:25.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:25.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:25.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:25.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:25.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:25] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:25.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:25] INFO grpc.py:275 (2): Received request to create bdev bdev_02593951-8f1d-487f-8486-78cf08ee9025 from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:25.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 5 2024-09-02T12:04:25.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:25.676787] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_02593951-8f1d-487f-8486-78cf08ee9025 rbd disk to lun 2024-09-02T12:04:25.918 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:25] INFO grpc.py:770 (2): Received request to add bdev_02593951-8f1d-487f-8486-78cf08ee9025 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:04:25.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:25.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:25] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:25.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:25] INFO grpc.py:275 (2): Received request to create bdev bdev_02593951-8f1d-487f-8486-78cf08ee9025 from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:25.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 5 2024-09-02T12:04:25.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:25.675729] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_02593951-8f1d-487f-8486-78cf08ee9025 rbd disk to lun 2024-09-02T12:04:25.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:25] INFO grpc.py:770 (2): Received request to add bdev_02593951-8f1d-487f-8486-78cf08ee9025 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:04:25.989 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:26.011 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:25] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:04:26.011 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:25] INFO grpc.py:275 (2): Received request to create bdev bdev_02593951-8f1d-487f-8486-78cf08ee9025 from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:26.011 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 5 2024-09-02T12:04:26.011 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:25.674399] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_02593951-8f1d-487f-8486-78cf08ee9025 rbd disk to lun 2024-09-02T12:04:26.011 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:25] INFO grpc.py:770 (2): Received request to add bdev_02593951-8f1d-487f-8486-78cf08ee9025 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:04:26.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:26.092 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:26.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:25] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 2, context: , client address: IPv4 172.21.15.38:32814 2024-09-02T12:04:26.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:25] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:26.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:25] INFO grpc.py:275 (2): Received request to create bdev bdev_02593951-8f1d-487f-8486-78cf08ee9025 from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:32814 2024-09-02T12:04:26.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 5 2024-09-02T12:04:26.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:25.648378] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_02593951-8f1d-487f-8486-78cf08ee9025 rbd disk to lun 2024-09-02T12:04:26.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:25] INFO grpc.py:770 (2): Received request to add bdev_02593951-8f1d-487f-8486-78cf08ee9025 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2, context: , client address: IPv4 172.21.15.38:32814 2024-09-02T12:04:26.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:26.193 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:26.225 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:26.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:26.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:26 smithi094 ceph-mon[20404]: pgmap v324: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.4 KiB/s wr, 61 op/s 2024-09-02T12:04:26.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:26 smithi087 ceph-mon[21100]: pgmap v324: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.4 KiB/s wr, 61 op/s 2024-09-02T12:04:26.508 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:26.508 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:26.508 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage18 2024-09-02T12:04:26.508 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage18 --load-balancing-group 3 2024-09-02T12:04:26.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:26 smithi064 ceph-mon[17163]: pgmap v324: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.4 KiB/s wr, 61 op/s 2024-09-02T12:04:26.596 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:26Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:27.308 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 18 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:27.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:27] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 3, context: , client address: IPv4 172.21.15.38:46060 2024-09-02T12:04:27.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:27] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:27.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:27] INFO grpc.py:275 (2): Received request to create bdev bdev_e304d724-9f62-459b-b789-7137b80a0618 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:46060 2024-09-02T12:04:27.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:27] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 5 2024-09-02T12:04:27.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:27.291494] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e304d724-9f62-459b-b789-7137b80a0618 rbd disk to lun 2024-09-02T12:04:27.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:27 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:27] INFO grpc.py:770 (2): Received request to add bdev_e304d724-9f62-459b-b789-7137b80a0618 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3, context: , client address: IPv4 172.21.15.38:46060 2024-09-02T12:04:27.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:27 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:27] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:27.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:27 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:27] INFO grpc.py:275 (2): Received request to create bdev bdev_e304d724-9f62-459b-b789-7137b80a0618 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:27.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:27 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:27] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 5 2024-09-02T12:04:27.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:27 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:27.321252] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e304d724-9f62-459b-b789-7137b80a0618 rbd disk to lun 2024-09-02T12:04:27.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:27 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:27] INFO grpc.py:770 (2): Received request to add bdev_e304d724-9f62-459b-b789-7137b80a0618 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:04:27.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:27 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:27] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:27.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:27 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:27] INFO grpc.py:275 (2): Received request to create bdev bdev_e304d724-9f62-459b-b789-7137b80a0618 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:27.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:27 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:27] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 5 2024-09-02T12:04:27.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:27 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:27.316452] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e304d724-9f62-459b-b789-7137b80a0618 rbd disk to lun 2024-09-02T12:04:27.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:27 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:27] INFO grpc.py:770 (2): Received request to add bdev_e304d724-9f62-459b-b789-7137b80a0618 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:04:27.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:27 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:27] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:04:27.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:27 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:27] INFO grpc.py:275 (2): Received request to create bdev bdev_e304d724-9f62-459b-b789-7137b80a0618 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:27.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:27 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:27] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 5 2024-09-02T12:04:27.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:27 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:27.318480] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e304d724-9f62-459b-b789-7137b80a0618 rbd disk to lun 2024-09-02T12:04:27.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:27 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:27] INFO grpc.py:770 (2): Received request to add bdev_e304d724-9f62-459b-b789-7137b80a0618 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:04:28.152 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:28.152 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:28.152 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage19 2024-09-02T12:04:28.152 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage19 --load-balancing-group 4 2024-09-02T12:04:28.235 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:28Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:28.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-mon[17163]: pgmap v325: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 2.0 KiB/s wr, 76 op/s 2024-09-02T12:04:28.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:28 smithi094 ceph-mon[20404]: pgmap v325: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 2.0 KiB/s wr, 76 op/s 2024-09-02T12:04:28.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:28 smithi087 ceph-mon[21100]: pgmap v325: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 2.0 KiB/s wr, 76 op/s 2024-09-02T12:04:28.973 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 19 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:29.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:28] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:29.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:28] INFO grpc.py:275 (2): Received request to create bdev bdev_e0592f88-f459-49b5-b269-949b89c731f7 from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:29.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 5 2024-09-02T12:04:29.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:28.978385] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e0592f88-f459-49b5-b269-949b89c731f7 rbd disk to lun 2024-09-02T12:04:29.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:28] INFO grpc.py:770 (2): Received request to add bdev_e0592f88-f459-49b5-b269-949b89c731f7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:04:29.265 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:28] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 4, context: , client address: IPv4 172.21.15.38:51540 2024-09-02T12:04:29.265 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:28] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:04:29.265 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:28] INFO grpc.py:275 (2): Received request to create bdev bdev_e0592f88-f459-49b5-b269-949b89c731f7 from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:51540 2024-09-02T12:04:29.265 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 5 2024-09-02T12:04:29.265 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:28.955956] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e0592f88-f459-49b5-b269-949b89c731f7 rbd disk to lun 2024-09-02T12:04:29.265 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:28] INFO grpc.py:770 (2): Received request to add bdev_e0592f88-f459-49b5-b269-949b89c731f7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4, context: , client address: IPv4 172.21.15.38:51540 2024-09-02T12:04:29.266 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:28] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:29.266 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:28] INFO grpc.py:275 (2): Received request to create bdev bdev_e0592f88-f459-49b5-b269-949b89c731f7 from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:29.266 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 5 2024-09-02T12:04:29.266 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:28.982593] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e0592f88-f459-49b5-b269-949b89c731f7 rbd disk to lun 2024-09-02T12:04:29.266 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:28] INFO grpc.py:770 (2): Received request to add bdev_e0592f88-f459-49b5-b269-949b89c731f7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:04:29.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:28] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:04:29.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:28] INFO grpc.py:275 (2): Received request to create bdev bdev_e0592f88-f459-49b5-b269-949b89c731f7 from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:29.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 5 2024-09-02T12:04:29.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:28.985355] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e0592f88-f459-49b5-b269-949b89c731f7 rbd disk to lun 2024-09-02T12:04:29.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:28] INFO grpc.py:770 (2): Received request to add bdev_e0592f88-f459-49b5-b269-949b89c731f7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:04:29.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:29 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2907646578' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:29.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:29 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2907646578' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:29.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:29 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2907646578' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:29.863 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:29.863 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:29.864 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage20 2024-09-02T12:04:29.864 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode1 --rbd-pool mypool --rbd-image myimage20 --load-balancing-group 1 2024-09-02T12:04:29.947 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:29Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-mon[17163]: pgmap v326: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 1.7 KiB/s wr, 88 op/s 2024-09-02T12:04:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:04:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:04:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:04:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:04:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-mon[20404]: pgmap v326: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 1.7 KiB/s wr, 88 op/s 2024-09-02T12:04:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:04:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:04:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:04:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:04:30.717 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 20 to nqn.2016-06.io.spdk:cnode1: Successful 2024-09-02T12:04:30.718 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-mon[21100]: pgmap v326: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 1.7 KiB/s wr, 88 op/s 2024-09-02T12:04:30.718 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:04:30.718 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:04:30.718 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:04:30.718 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:04:31.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:30] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:31.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:30] INFO grpc.py:275 (2): Received request to create bdev bdev_42eb2216-ff26-4390-90a3-97b35df81f87 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:31.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 5 2024-09-02T12:04:31.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:30.727501] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 rbd disk to lun 2024-09-02T12:04:31.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:30] INFO grpc.py:770 (2): Received request to add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:04:31.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:30] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode1, ana group 1, context: , client address: IPv4 172.21.15.38:46652 2024-09-02T12:04:31.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:30] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:31.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:30] INFO grpc.py:275 (2): Received request to create bdev bdev_42eb2216-ff26-4390-90a3-97b35df81f87 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:46652 2024-09-02T12:04:31.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 5 2024-09-02T12:04:31.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:30.700433] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 rbd disk to lun 2024-09-02T12:04:31.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:30] INFO grpc.py:770 (2): Received request to add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1, context: , client address: IPv4 172.21.15.38:46652 2024-09-02T12:04:31.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:30] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:31.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:30] INFO grpc.py:275 (2): Received request to create bdev bdev_42eb2216-ff26-4390-90a3-97b35df81f87 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:31.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 5 2024-09-02T12:04:31.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:30.725915] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 rbd disk to lun 2024-09-02T12:04:31.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:30] INFO grpc.py:770 (2): Received request to add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:04:31.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:30] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:04:31.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:30] INFO grpc.py:275 (2): Received request to create bdev bdev_42eb2216-ff26-4390-90a3-97b35df81f87 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:31.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 5 2024-09-02T12:04:31.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:30.731501] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 rbd disk to lun 2024-09-02T12:04:31.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:30] INFO grpc.py:770 (2): Received request to add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:04:31.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:31.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:31.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:31.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:31.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:31.565 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:31.566 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:04:31.566 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:04:31.566 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 20 2024-09-02T12:04:31.567 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:31.567 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage21 2024-09-02T12:04:31.567 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage21 --load-balancing-group 2 2024-09-02T12:04:31.568 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:31.600 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:31.656 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:31Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:31.671 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:31.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:31.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:31.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:31.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:32.445 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 1 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:32.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-mon[17163]: pgmap v327: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 2.0 KiB/s wr, 75 op/s 2024-09-02T12:04:32.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:32] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, context: , client address: IPv4 172.21.15.38:42564 2024-09-02T12:04:32.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:32] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:32.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:32] INFO grpc.py:275 (2): Received request to create bdev bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:42564 2024-09-02T12:04:32.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:32] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 6 2024-09-02T12:04:32.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:32.429026] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 rbd disk to lun 2024-09-02T12:04:32.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:32 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:32] INFO grpc.py:770 (2): Received request to add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2, context: , client address: IPv4 172.21.15.38:42564 2024-09-02T12:04:32.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:32 smithi094 ceph-mon[20404]: pgmap v327: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 2.0 KiB/s wr, 75 op/s 2024-09-02T12:04:32.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:32] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:32.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:32] INFO grpc.py:275 (2): Received request to create bdev bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:32.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:32] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 6 2024-09-02T12:04:32.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:32.455034] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 rbd disk to lun 2024-09-02T12:04:32.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:32] INFO grpc.py:770 (2): Received request to add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:04:32.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:32] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:32.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:32] INFO grpc.py:275 (2): Received request to create bdev bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:32.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:32] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 6 2024-09-02T12:04:32.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:32.452702] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 rbd disk to lun 2024-09-02T12:04:32.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:32] INFO grpc.py:770 (2): Received request to add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:04:32.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:32 smithi087 ceph-mon[21100]: pgmap v327: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 2.0 KiB/s wr, 75 op/s 2024-09-02T12:04:32.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:32] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:32.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:32] INFO grpc.py:275 (2): Received request to create bdev bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:32.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:32] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 6 2024-09-02T12:04:32.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:32.456860] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 rbd disk to lun 2024-09-02T12:04:32.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:32] INFO grpc.py:770 (2): Received request to add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:04:33.300 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:33.300 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:33.300 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage22 2024-09-02T12:04:33.300 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage22 --load-balancing-group 3 2024-09-02T12:04:33.392 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:33Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:34.184 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 2 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:34.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:34 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:34] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:34.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:34 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:34] INFO grpc.py:275 (2): Received request to create bdev bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:34.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:34 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:34] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 6 2024-09-02T12:04:34.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:34 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:34.193625] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 rbd disk to lun 2024-09-02T12:04:34.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:34 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:34] INFO grpc.py:770 (2): Received request to add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:04:34.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:34 smithi087 ceph-mon[21100]: pgmap v328: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:04:34.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:34] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:34.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:34] INFO grpc.py:275 (2): Received request to create bdev bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:34.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:34] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 6 2024-09-02T12:04:34.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:34.194928] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 rbd disk to lun 2024-09-02T12:04:34.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:34 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:34] INFO grpc.py:770 (2): Received request to add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:04:34.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-mon[17163]: pgmap v328: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:04:34.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:34] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, context: , client address: IPv4 172.21.15.38:60546 2024-09-02T12:04:34.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:34] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:34.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:34] INFO grpc.py:275 (2): Received request to create bdev bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:60546 2024-09-02T12:04:34.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:34] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 6 2024-09-02T12:04:34.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:34.168158] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 rbd disk to lun 2024-09-02T12:04:34.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:34 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:34] INFO grpc.py:770 (2): Received request to add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3, context: , client address: IPv4 172.21.15.38:60546 2024-09-02T12:04:34.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:34 smithi094 ceph-mon[20404]: pgmap v328: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:04:34.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:34] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:34.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:34] INFO grpc.py:275 (2): Received request to create bdev bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:34.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:34] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 6 2024-09-02T12:04:34.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:34.197546] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 rbd disk to lun 2024-09-02T12:04:34.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:34 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:34] INFO grpc.py:770 (2): Received request to add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:04:35.077 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:35.077 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:35.077 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage23 2024-09-02T12:04:35.077 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage23 --load-balancing-group 4 2024-09-02T12:04:35.167 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:35Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:35.907 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 3 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:36.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:35] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:36.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:35] INFO grpc.py:275 (2): Received request to create bdev bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:36.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 6 2024-09-02T12:04:36.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:35.919873] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f rbd disk to lun 2024-09-02T12:04:36.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:35] INFO grpc.py:770 (2): Received request to add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:04:36.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:35] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:36.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:35] INFO grpc.py:275 (2): Received request to create bdev bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:36.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 6 2024-09-02T12:04:36.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:35.915506] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f rbd disk to lun 2024-09-02T12:04:36.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:35] INFO grpc.py:770 (2): Received request to add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:04:36.290 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:35] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, context: , client address: IPv4 172.21.15.38:33608 2024-09-02T12:04:36.290 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:35] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:36.290 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:35] INFO grpc.py:275 (2): Received request to create bdev bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:33608 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 6 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:35.892049] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f rbd disk to lun 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:35] INFO grpc.py:770 (2): Received request to add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f to nqn.2016-06.io.spdk:cnode2 with ANA group id 4, context: , client address: IPv4 172.21.15.38:33608 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:35] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:35] INFO grpc.py:275 (2): Received request to create bdev bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 6 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:35.918861] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f rbd disk to lun 2024-09-02T12:04:36.291 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:35] INFO grpc.py:770 (2): Received request to add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:04:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:36 smithi064 ceph-mon[17163]: pgmap v329: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 1.6 KiB/s wr, 77 op/s 2024-09-02T12:04:36.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:36 smithi094 ceph-mon[20404]: pgmap v329: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 1.6 KiB/s wr, 77 op/s 2024-09-02T12:04:36.721 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:36.721 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:36.721 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage24 2024-09-02T12:04:36.721 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage24 --load-balancing-group 1 2024-09-02T12:04:36.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:36 smithi087 ceph-mon[21100]: pgmap v329: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 1.6 KiB/s wr, 77 op/s 2024-09-02T12:04:36.805 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:36Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:36.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:36.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:36.965 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:36.997 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:37.068 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:37.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:37.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:37.240 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:37.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:37.344 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:37.374 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:37.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:37] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, context: , client address: IPv4 172.21.15.38:40760 2024-09-02T12:04:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:37] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:37] INFO grpc.py:275 (2): Received request to create bdev bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:40760 2024-09-02T12:04:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 6 2024-09-02T12:04:37.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:37.576646] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 rbd disk to lun 2024-09-02T12:04:37.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:37] INFO grpc.py:770 (2): Received request to add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1, context: , client address: IPv4 172.21.15.38:40760 2024-09-02T12:04:37.595 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 4 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:37.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:37] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:37.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:37] INFO grpc.py:275 (2): Received request to create bdev bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:37.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 6 2024-09-02T12:04:37.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:37.604125] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 rbd disk to lun 2024-09-02T12:04:37.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:37] INFO grpc.py:770 (2): Received request to add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:04:37.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:37] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:37.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:37] INFO grpc.py:275 (2): Received request to create bdev bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:37.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 6 2024-09-02T12:04:37.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:37.608423] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 rbd disk to lun 2024-09-02T12:04:37.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:37] INFO grpc.py:770 (2): Received request to add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:04:37.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:37] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:37.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:37] INFO grpc.py:275 (2): Received request to create bdev bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:37.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 6 2024-09-02T12:04:37.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:37.602083] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 rbd disk to lun 2024-09-02T12:04:37.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:37] INFO grpc.py:770 (2): Received request to add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:04:38.505 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:38.505 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:38.505 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage25 2024-09-02T12:04:38.505 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage25 --load-balancing-group 2 2024-09-02T12:04:38.583 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:38Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:38.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:38 smithi064 ceph-mon[17163]: pgmap v330: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 2.1 KiB/s wr, 84 op/s 2024-09-02T12:04:38.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:38 smithi094 ceph-mon[20404]: pgmap v330: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 2.1 KiB/s wr, 84 op/s 2024-09-02T12:04:38.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:38 smithi087 ceph-mon[21100]: pgmap v330: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 2.1 KiB/s wr, 84 op/s 2024-09-02T12:04:39.313 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 5 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:39.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/934568772' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:39.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:39] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, context: , client address: IPv4 172.21.15.38:48546 2024-09-02T12:04:39.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:39] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:04:39.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:39] INFO grpc.py:275 (2): Received request to create bdev bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:48546 2024-09-02T12:04:39.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:39] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 7 2024-09-02T12:04:39.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:39.297014] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc rbd disk to lun 2024-09-02T12:04:39.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:39 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:39] INFO grpc.py:770 (2): Received request to add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc to nqn.2016-06.io.spdk:cnode2 with ANA group id 2, context: , client address: IPv4 172.21.15.38:48546 2024-09-02T12:04:39.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:39 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/934568772' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:39.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:39 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:39] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:39.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:39 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:39] INFO grpc.py:275 (2): Received request to create bdev bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:39.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:39 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:39] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 7 2024-09-02T12:04:39.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:39 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:39.326905] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc rbd disk to lun 2024-09-02T12:04:39.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:39 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:39] INFO grpc.py:770 (2): Received request to add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:04:39.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:39 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:39] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:39.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:39 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:39] INFO grpc.py:275 (2): Received request to create bdev bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:39.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:39 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:39] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 7 2024-09-02T12:04:39.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:39 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:39.323676] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc rbd disk to lun 2024-09-02T12:04:39.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:39 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:39] INFO grpc.py:770 (2): Received request to add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:04:39.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:39 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/934568772' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:39.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:39 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:39] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:39.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:39 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:39] INFO grpc.py:275 (2): Received request to create bdev bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:39.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:39 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:39] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 7 2024-09-02T12:04:39.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:39 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:39.325457] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc rbd disk to lun 2024-09-02T12:04:39.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:39 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:39] INFO grpc.py:770 (2): Received request to add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:04:40.215 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:40.216 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:40.216 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage26 2024-09-02T12:04:40.216 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage26 --load-balancing-group 3 2024-09-02T12:04:40.292 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:40Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:40.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:40 smithi064 ceph-mon[17163]: pgmap v331: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:04:40.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:40 smithi094 ceph-mon[20404]: pgmap v331: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:04:40.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:40 smithi087 ceph-mon[21100]: pgmap v331: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:04:41.050 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 6 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:41.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:41] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, context: , client address: IPv4 172.21.15.38:57282 2024-09-02T12:04:41.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:41] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:41.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:41] INFO grpc.py:275 (2): Received request to create bdev bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:57282 2024-09-02T12:04:41.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:41] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 7 2024-09-02T12:04:41.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:41.034559] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f rbd disk to lun 2024-09-02T12:04:41.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:41 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:41] INFO grpc.py:770 (2): Received request to add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3, context: , client address: IPv4 172.21.15.38:57282 2024-09-02T12:04:41.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:41 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:41] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:41.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:41 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:41] INFO grpc.py:275 (2): Received request to create bdev bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:41.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:41 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:41] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 7 2024-09-02T12:04:41.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:41 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:41.059747] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f rbd disk to lun 2024-09-02T12:04:41.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:41 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:41] INFO grpc.py:770 (2): Received request to add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:04:41.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:41 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:41] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:41.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:41 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:41] INFO grpc.py:275 (2): Received request to create bdev bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:41.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:41 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:41] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 7 2024-09-02T12:04:41.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:41 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:41.060982] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f rbd disk to lun 2024-09-02T12:04:41.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:41 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:41] INFO grpc.py:770 (2): Received request to add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:04:41.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:41 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:41] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:41.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:41 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:41] INFO grpc.py:275 (2): Received request to create bdev bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:41.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:41 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:41] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 7 2024-09-02T12:04:41.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:41 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:41.063391] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f rbd disk to lun 2024-09-02T12:04:41.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:41 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:41] INFO grpc.py:770 (2): Received request to add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:04:41.950 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:41.950 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:41.950 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage27 2024-09-02T12:04:41.951 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage27 --load-balancing-group 4 2024-09-02T12:04:42.031 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:42Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:42.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:42.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:42.537 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:42.567 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:42 smithi087 ceph-mon[21100]: pgmap v332: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 2.1 KiB/s wr, 83 op/s 2024-09-02T12:04:42.568 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:42.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-mon[17163]: pgmap v332: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 2.1 KiB/s wr, 83 op/s 2024-09-02T12:04:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:42 smithi094 ceph-mon[20404]: pgmap v332: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 2.1 KiB/s wr, 83 op/s 2024-09-02T12:04:42.640 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:42.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:42.744 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:42.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:42.818 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 7 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:42.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:42.911 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:42] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, context: , client address: IPv4 172.21.15.38:38376 2024-09-02T12:04:42.911 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:42] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:42.911 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:42] INFO grpc.py:275 (2): Received request to create bdev bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:38376 2024-09-02T12:04:42.911 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:42] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 7 2024-09-02T12:04:42.911 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:42.801938] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 rbd disk to lun 2024-09-02T12:04:42.911 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:42 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:42] INFO grpc.py:770 (2): Received request to add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4, context: , client address: IPv4 172.21.15.38:38376 2024-09-02T12:04:42.920 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:42.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:42.982 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:42 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:42] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:42.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:42 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:42] INFO grpc.py:275 (2): Received request to create bdev bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:42.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:42 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:42] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 7 2024-09-02T12:04:42.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:42 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:42.829451] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 rbd disk to lun 2024-09-02T12:04:42.983 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:42 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:42] INFO grpc.py:770 (2): Received request to add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:04:42.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:43.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:42 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:42] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:43.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:42 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:42] INFO grpc.py:275 (2): Received request to create bdev bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:43.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:42 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:42] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 7 2024-09-02T12:04:43.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:42 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:42.826779] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 rbd disk to lun 2024-09-02T12:04:43.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:42 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:42] INFO grpc.py:770 (2): Received request to add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:04:43.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:42] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:43.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:42] INFO grpc.py:275 (2): Received request to create bdev bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:43.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:42] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 7 2024-09-02T12:04:43.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:42.828422] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 rbd disk to lun 2024-09-02T12:04:43.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:42] INFO grpc.py:770 (2): Received request to add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:04:43.653 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:43.653 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:43.653 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage28 2024-09-02T12:04:43.653 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage28 --load-balancing-group 1 2024-09-02T12:04:43.724 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:43Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:44.454 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 8 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-mon[17163]: pgmap v333: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:04:44.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:44] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, context: , client address: IPv4 172.21.15.38:44240 2024-09-02T12:04:44.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:44] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:44.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:44] INFO grpc.py:275 (2): Received request to create bdev bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:44240 2024-09-02T12:04:44.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:44] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 7 2024-09-02T12:04:44.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:44.437937] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d rbd disk to lun 2024-09-02T12:04:44.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:44 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:44] INFO grpc.py:770 (2): Received request to add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d to nqn.2016-06.io.spdk:cnode2 with ANA group id 1, context: , client address: IPv4 172.21.15.38:44240 2024-09-02T12:04:44.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:44 smithi094 ceph-mon[20404]: pgmap v333: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:04:44.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:44] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:44.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:44] INFO grpc.py:275 (2): Received request to create bdev bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:44.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:44] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 7 2024-09-02T12:04:44.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:44.467600] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d rbd disk to lun 2024-09-02T12:04:44.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:44 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:44] INFO grpc.py:770 (2): Received request to add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:04:44.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:44 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:44] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:44.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:44 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:44] INFO grpc.py:275 (2): Received request to create bdev bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:44.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:44 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:44] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 7 2024-09-02T12:04:44.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:44 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:44.464569] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d rbd disk to lun 2024-09-02T12:04:44.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:44 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:44] INFO grpc.py:770 (2): Received request to add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:04:44.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:44 smithi087 ceph-mon[21100]: pgmap v333: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:04:44.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:44] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:44.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:44] INFO grpc.py:275 (2): Received request to create bdev bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:44.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:44] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 7 2024-09-02T12:04:44.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:44.466514] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d rbd disk to lun 2024-09-02T12:04:44.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:44] INFO grpc.py:770 (2): Received request to add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:04:45.354 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:45.354 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:45.354 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage29 2024-09-02T12:04:45.354 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage29 --load-balancing-group 2 2024-09-02T12:04:45.454 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:45Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:46.219 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 9 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:46.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:46 smithi087 ceph-mon[21100]: pgmap v334: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:04:46.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:46 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:46] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:46.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:46 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:46] INFO grpc.py:275 (2): Received request to create bdev bdev_88e9aca1-c840-4d93-9444-7a339bb07efb from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:46.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:46 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:46] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 8 2024-09-02T12:04:46.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:46 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:46.231832] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb rbd disk to lun 2024-09-02T12:04:46.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:46 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:46] INFO grpc.py:770 (2): Received request to add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:04:46.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-mon[17163]: pgmap v334: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:04:46.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:46] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, context: , client address: IPv4 172.21.15.38:58136 2024-09-02T12:04:46.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:46] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:46.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:46] INFO grpc.py:275 (2): Received request to create bdev bdev_88e9aca1-c840-4d93-9444-7a339bb07efb from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:58136 2024-09-02T12:04:46.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:46] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 8 2024-09-02T12:04:46.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:46.204023] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb rbd disk to lun 2024-09-02T12:04:46.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:46 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:46] INFO grpc.py:770 (2): Received request to add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb to nqn.2016-06.io.spdk:cnode2 with ANA group id 2, context: , client address: IPv4 172.21.15.38:58136 2024-09-02T12:04:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:46 smithi094 ceph-mon[20404]: pgmap v334: 33 pgs: 33 active+clean; 609 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:04:46.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:46 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:46] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:46.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:46 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:46] INFO grpc.py:275 (2): Received request to create bdev bdev_88e9aca1-c840-4d93-9444-7a339bb07efb from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:46.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:46 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:46] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 8 2024-09-02T12:04:46.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:46 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:46.227675] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb rbd disk to lun 2024-09-02T12:04:46.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:46 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:46] INFO grpc.py:770 (2): Received request to add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:04:46.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:46 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:46] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:46.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:46 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:46] INFO grpc.py:275 (2): Received request to create bdev bdev_88e9aca1-c840-4d93-9444-7a339bb07efb from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:46.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:46 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:46] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 8 2024-09-02T12:04:46.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:46 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:46.229298] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb rbd disk to lun 2024-09-02T12:04:46.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:46 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:46] INFO grpc.py:770 (2): Received request to add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:04:47.057 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:47.057 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:47.057 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage30 2024-09-02T12:04:47.057 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage30 --load-balancing-group 3 2024-09-02T12:04:47.144 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:47Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:47.872 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 10 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:48.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:48.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:48.106 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:47] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, context: , client address: IPv4 172.21.15.38:43046 2024-09-02T12:04:48.106 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:47] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:48.107 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:47] INFO grpc.py:275 (2): Received request to create bdev bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:43046 2024-09-02T12:04:48.107 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:47] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 8 2024-09-02T12:04:48.107 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:47.855572] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 rbd disk to lun 2024-09-02T12:04:48.107 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:47] INFO grpc.py:770 (2): Received request to add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3, context: , client address: IPv4 172.21.15.38:43046 2024-09-02T12:04:48.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:48.137 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:47 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:47] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:48.137 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:47 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:47] INFO grpc.py:275 (2): Received request to create bdev bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:48.137 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:47 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:47] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 8 2024-09-02T12:04:48.137 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:47 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:47.879349] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 rbd disk to lun 2024-09-02T12:04:48.137 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:47 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:47] INFO grpc.py:770 (2): Received request to add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:04:48.147 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:48.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:47 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:47] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:48.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:47 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:47] INFO grpc.py:275 (2): Received request to create bdev bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:48.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:47 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:47] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 8 2024-09-02T12:04:48.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:47 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:47.877308] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 rbd disk to lun 2024-09-02T12:04:48.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:47 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:47] INFO grpc.py:770 (2): Received request to add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:04:48.219 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:48.290 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:48.311 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:47] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:48.311 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:47] INFO grpc.py:275 (2): Received request to create bdev bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:48.311 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:47] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 8 2024-09-02T12:04:48.311 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:47.883757] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 rbd disk to lun 2024-09-02T12:04:48.311 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:47] INFO grpc.py:770 (2): Received request to add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:04:48.322 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:48.394 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:48.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:48.497 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:48.499 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:48 smithi087 ceph-mon[21100]: pgmap v335: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 2.0 KiB/s wr, 77 op/s 2024-09-02T12:04:48.527 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:48.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:48 smithi094 ceph-mon[20404]: pgmap v335: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 2.0 KiB/s wr, 77 op/s 2024-09-02T12:04:48.734 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:48.734 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:48.734 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage31 2024-09-02T12:04:48.734 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage31 --load-balancing-group 4 2024-09-02T12:04:48.815 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:48Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:48.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:48 smithi064 ceph-mon[17163]: pgmap v335: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 2.0 KiB/s wr, 77 op/s 2024-09-02T12:04:49.545 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 11 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:49.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:49] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, context: , client address: IPv4 172.21.15.38:47270 2024-09-02T12:04:49.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:49] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:04:49.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:49] INFO grpc.py:275 (2): Received request to create bdev bdev_2b28c17b-650d-4aad-aa11-b432947561d5 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:47270 2024-09-02T12:04:49.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:49] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 8 2024-09-02T12:04:49.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:49.527778] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 rbd disk to lun 2024-09-02T12:04:49.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:49 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:49] INFO grpc.py:770 (2): Received request to add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4, context: , client address: IPv4 172.21.15.38:47270 2024-09-02T12:04:49.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:49] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:49.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:49] INFO grpc.py:275 (2): Received request to create bdev bdev_2b28c17b-650d-4aad-aa11-b432947561d5 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:49.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:49] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 8 2024-09-02T12:04:49.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:49.555163] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 rbd disk to lun 2024-09-02T12:04:49.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:49] INFO grpc.py:770 (2): Received request to add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:04:49.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:49 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:49] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:49.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:49 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:49] INFO grpc.py:275 (2): Received request to create bdev bdev_2b28c17b-650d-4aad-aa11-b432947561d5 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:49.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:49 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:49] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 8 2024-09-02T12:04:49.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:49 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:49.551117] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 rbd disk to lun 2024-09-02T12:04:49.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:49 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:49] INFO grpc.py:770 (2): Received request to add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:04:50.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:49] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:50.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:49] INFO grpc.py:275 (2): Received request to create bdev bdev_2b28c17b-650d-4aad-aa11-b432947561d5 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:50.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:49] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 8 2024-09-02T12:04:50.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:49.552829] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 rbd disk to lun 2024-09-02T12:04:50.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:49] INFO grpc.py:770 (2): Received request to add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:04:50.444 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:50.444 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:50.444 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage32 2024-09-02T12:04:50.444 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage32 --load-balancing-group 1 2024-09-02T12:04:50.528 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:50Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:50 smithi087 ceph-mon[21100]: pgmap v336: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 1.6 KiB/s wr, 80 op/s 2024-09-02T12:04:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:50 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3616501283' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:50.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:50 smithi064 ceph-mon[17163]: pgmap v336: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 1.6 KiB/s wr, 80 op/s 2024-09-02T12:04:50.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:50 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3616501283' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:50.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:50 smithi094 ceph-mon[20404]: pgmap v336: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 1.6 KiB/s wr, 80 op/s 2024-09-02T12:04:50.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:50 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3616501283' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:04:51.275 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 12 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:51.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:51] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, context: , client address: IPv4 172.21.15.38:46168 2024-09-02T12:04:51.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:51] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:51.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:51] INFO grpc.py:275 (2): Received request to create bdev bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:46168 2024-09-02T12:04:51.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:51] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 8 2024-09-02T12:04:51.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:51.256756] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 rbd disk to lun 2024-09-02T12:04:51.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:51 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:51] INFO grpc.py:770 (2): Received request to add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1, context: , client address: IPv4 172.21.15.38:46168 2024-09-02T12:04:51.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:51] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:51.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:51] INFO grpc.py:275 (2): Received request to create bdev bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:51.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:51] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 8 2024-09-02T12:04:51.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:51.283245] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 rbd disk to lun 2024-09-02T12:04:51.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:51] INFO grpc.py:770 (2): Received request to add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:04:51.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:51 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:51] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:51.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:51 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:51] INFO grpc.py:275 (2): Received request to create bdev bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:51.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:51 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:51] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 8 2024-09-02T12:04:51.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:51 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:51.288403] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 rbd disk to lun 2024-09-02T12:04:51.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:51 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:51] INFO grpc.py:770 (2): Received request to add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:04:51.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:51] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:51.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:51] INFO grpc.py:275 (2): Received request to create bdev bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:51.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:51] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 8 2024-09-02T12:04:51.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:51.285688] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 rbd disk to lun 2024-09-02T12:04:51.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:51] INFO grpc.py:770 (2): Received request to add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:04:52.186 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:52.186 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:52.186 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage33 2024-09-02T12:04:52.186 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage33 --load-balancing-group 2 2024-09-02T12:04:52.262 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:52Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:52.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:52 smithi087 ceph-mon[21100]: pgmap v337: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:04:52.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-mon[17163]: pgmap v337: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:04:52.848 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:52 smithi094 ceph-mon[20404]: pgmap v337: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:04:52.978 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 13 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:53.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:52] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:53.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:52] INFO grpc.py:275 (2): Received request to create bdev bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:53.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:52] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 9 2024-09-02T12:04:53.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:52.990915] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 rbd disk to lun 2024-09-02T12:04:53.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:52] INFO grpc.py:770 (2): Received request to add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:04:53.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:52] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, context: , client address: IPv4 172.21.15.38:41496 2024-09-02T12:04:53.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:52] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:53.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:52] INFO grpc.py:275 (2): Received request to create bdev bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:41496 2024-09-02T12:04:53.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:52] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 9 2024-09-02T12:04:53.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:52.962549] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 rbd disk to lun 2024-09-02T12:04:53.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:52 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:52] INFO grpc.py:770 (2): Received request to add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2, context: , client address: IPv4 172.21.15.38:41496 2024-09-02T12:04:53.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:52] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:53.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:52] INFO grpc.py:275 (2): Received request to create bdev bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:53.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:52] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 9 2024-09-02T12:04:53.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:52.986875] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 rbd disk to lun 2024-09-02T12:04:53.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:52] INFO grpc.py:770 (2): Received request to add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:04:53.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:52 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:52] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:04:53.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:52 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:52] INFO grpc.py:275 (2): Received request to create bdev bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:53.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:52 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:52] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 9 2024-09-02T12:04:53.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:52 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:52.990306] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 rbd disk to lun 2024-09-02T12:04:53.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:52 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:52] INFO grpc.py:770 (2): Received request to add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:04:53.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:53.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:53.693 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:53.724 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:53.795 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:53.866 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:53.872 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:53.872 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:53.873 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage34 2024-09-02T12:04:53.873 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage34 --load-balancing-group 3 2024-09-02T12:04:53.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:53.962 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:53Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:53.972 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:54.046 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:54.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:54.108 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:54.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:54.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:54 smithi094 ceph-mon[20404]: pgmap v338: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 1.6 KiB/s wr, 80 op/s 2024-09-02T12:04:54.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-mon[17163]: pgmap v338: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 1.6 KiB/s wr, 80 op/s 2024-09-02T12:04:54.709 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 14 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:54.709 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:54 smithi087 ceph-mon[21100]: pgmap v338: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 1.6 KiB/s wr, 80 op/s 2024-09-02T12:04:55.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:54 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:54] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:55.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:54 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:54] INFO grpc.py:275 (2): Received request to create bdev bdev_a995fbc3-9dbc-43b9-ab21-579813796890 from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:55.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:54 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:54] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 9 2024-09-02T12:04:55.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:54 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:54.720824] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 rbd disk to lun 2024-09-02T12:04:55.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:54 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:54] INFO grpc.py:770 (2): Received request to add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:04:55.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:54] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, context: , client address: IPv4 172.21.15.38:40246 2024-09-02T12:04:55.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:54] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:55.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:54] INFO grpc.py:275 (2): Received request to create bdev bdev_a995fbc3-9dbc-43b9-ab21-579813796890 from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:40246 2024-09-02T12:04:55.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:54] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 9 2024-09-02T12:04:55.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:54.691373] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 rbd disk to lun 2024-09-02T12:04:55.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:54 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:54] INFO grpc.py:770 (2): Received request to add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3, context: , client address: IPv4 172.21.15.38:40246 2024-09-02T12:04:55.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:54] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:55.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:54] INFO grpc.py:275 (2): Received request to create bdev bdev_a995fbc3-9dbc-43b9-ab21-579813796890 from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:55.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:54] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 9 2024-09-02T12:04:55.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:54.719395] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 rbd disk to lun 2024-09-02T12:04:55.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:54] INFO grpc.py:770 (2): Received request to add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:04:55.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:54 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:54] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:04:55.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:54 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:54] INFO grpc.py:275 (2): Received request to create bdev bdev_a995fbc3-9dbc-43b9-ab21-579813796890 from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:55.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:54 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:54] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 9 2024-09-02T12:04:55.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:54 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:54.721709] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 rbd disk to lun 2024-09-02T12:04:55.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:54 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:54] INFO grpc.py:770 (2): Received request to add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:04:55.533 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:55.533 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:55.533 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage35 2024-09-02T12:04:55.533 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage35 --load-balancing-group 4 2024-09-02T12:04:55.628 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:55Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:55.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:55 smithi087 ceph-mon[21100]: pgmap v339: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:04:55.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:55 smithi064 ceph-mon[17163]: pgmap v339: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:04:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:55 smithi094 ceph-mon[20404]: pgmap v339: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:04:56.377 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 15 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:56.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:56 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:56] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:56.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:56 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:56] INFO grpc.py:275 (2): Received request to create bdev bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:56.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:56 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:56] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 9 2024-09-02T12:04:56.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:56 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:56.387407] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 rbd disk to lun 2024-09-02T12:04:56.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:56 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:56] INFO grpc.py:770 (2): Received request to add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:04:56.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:56] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:56.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:56] INFO grpc.py:275 (2): Received request to create bdev bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:56.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:56] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 9 2024-09-02T12:04:56.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:56.385893] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 rbd disk to lun 2024-09-02T12:04:56.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:56] INFO grpc.py:770 (2): Received request to add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:04:56.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:56] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, context: , client address: IPv4 172.21.15.38:51034 2024-09-02T12:04:56.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:56] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:56.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:56] INFO grpc.py:275 (2): Received request to create bdev bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:51034 2024-09-02T12:04:56.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:56] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 9 2024-09-02T12:04:56.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:56.360784] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 rbd disk to lun 2024-09-02T12:04:56.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:56 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:56] INFO grpc.py:770 (2): Received request to add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4, context: , client address: IPv4 172.21.15.38:51034 2024-09-02T12:04:56.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:56] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:04:56.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:56] INFO grpc.py:275 (2): Received request to create bdev bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:56.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:56] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 9 2024-09-02T12:04:56.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:56.383732] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 rbd disk to lun 2024-09-02T12:04:56.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:56] INFO grpc.py:770 (2): Received request to add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:04:57.261 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:57.261 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:57.261 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage36 2024-09-02T12:04:57.261 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage36 --load-balancing-group 1 2024-09-02T12:04:57.347 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:57Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:57.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:57 smithi087 ceph-mon[21100]: pgmap v340: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 1.8 KiB/s wr, 73 op/s 2024-09-02T12:04:57.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:57 smithi064 ceph-mon[17163]: pgmap v340: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 1.8 KiB/s wr, 73 op/s 2024-09-02T12:04:57.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:57 smithi094 ceph-mon[20404]: pgmap v340: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 1.8 KiB/s wr, 73 op/s 2024-09-02T12:04:58.089 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 16 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:04:58.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:58] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, context: , client address: IPv4 172.21.15.38:45862 2024-09-02T12:04:58.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:58] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:04:58.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:58] INFO grpc.py:275 (2): Received request to create bdev bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:45862 2024-09-02T12:04:58.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 9 2024-09-02T12:04:58.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:58.072585] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 rbd disk to lun 2024-09-02T12:04:58.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:58 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:58] INFO grpc.py:770 (2): Received request to add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1, context: , client address: IPv4 172.21.15.38:45862 2024-09-02T12:04:58.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:58] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:58.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:58] INFO grpc.py:275 (2): Received request to create bdev bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:58.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 9 2024-09-02T12:04:58.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:58.095538] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 rbd disk to lun 2024-09-02T12:04:58.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:58] INFO grpc.py:770 (2): Received request to add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:04:58.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:58] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:58.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:58] INFO grpc.py:275 (2): Received request to create bdev bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:58.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 9 2024-09-02T12:04:58.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:58.101145] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 rbd disk to lun 2024-09-02T12:04:58.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:58] INFO grpc.py:770 (2): Received request to add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:04:58.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:58] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:04:58.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:58] INFO grpc.py:275 (2): Received request to create bdev bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:04:58.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:58] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 9 2024-09-02T12:04:58.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:58.097898] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 rbd disk to lun 2024-09-02T12:04:58.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:58 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:58] INFO grpc.py:770 (2): Received request to add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:04:58.930 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:04:58.930 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:04:58.930 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage37 2024-09-02T12:04:58.930 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage37 --load-balancing-group 2 2024-09-02T12:04:59.023 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:04:59Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:04:59.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:04:59.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:04:59.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:04:59.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:04:59.376 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:04:59.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:04:59.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:04:59.551 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:04:59.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-mon[17163]: pgmap v341: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 1.4 KiB/s wr, 75 op/s 2024-09-02T12:04:59.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:04:59.654 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:04:59.675 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:04:59 smithi087 ceph-mon[21100]: pgmap v341: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 1.4 KiB/s wr, 75 op/s 2024-09-02T12:04:59.685 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:04:59.706 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:04:59 smithi094 ceph-mon[20404]: pgmap v341: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 1.4 KiB/s wr, 75 op/s 2024-09-02T12:04:59.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:04:59.737 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 17 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:05:00.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:59 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:59] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:05:00.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:59 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:59] INFO grpc.py:275 (2): Received request to create bdev bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:00.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:59 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:59] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 10 2024-09-02T12:05:00.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:59 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:04:59.746806] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 rbd disk to lun 2024-09-02T12:05:00.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:04:59 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:04:59] INFO grpc.py:770 (2): Received request to add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:05:00.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:59] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 2, context: , client address: IPv4 172.21.15.38:38592 2024-09-02T12:05:00.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:59] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:05:00.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:59] INFO grpc.py:275 (2): Received request to create bdev bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:38592 2024-09-02T12:05:00.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:59] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 10 2024-09-02T12:05:00.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:04:59.720660] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 rbd disk to lun 2024-09-02T12:05:00.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:04:59 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:04:59] INFO grpc.py:770 (2): Received request to add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2, context: , client address: IPv4 172.21.15.38:38592 2024-09-02T12:05:00.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:59] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:05:00.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:59] INFO grpc.py:275 (2): Received request to create bdev bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:00.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:59] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 10 2024-09-02T12:05:00.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:04:59.743674] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 rbd disk to lun 2024-09-02T12:05:00.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:04:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:04:59] INFO grpc.py:770 (2): Received request to add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:05:00.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:59 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:59] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:05:00.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:59 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:59] INFO grpc.py:275 (2): Received request to create bdev bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:00.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:59 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:59] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 10 2024-09-02T12:05:00.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:59 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:04:59.747592] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 rbd disk to lun 2024-09-02T12:05:00.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:04:59 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:04:59] INFO grpc.py:770 (2): Received request to add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:05:00.572 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:00.573 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:00.573 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage38 2024-09-02T12:05:00.573 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage38 --load-balancing-group 3 2024-09-02T12:05:00.651 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:00Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:00.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:00 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/959590654' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:00.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:00 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/959590654' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:00.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:00 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/959590654' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:01.407 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 18 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:05:01.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:01 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:01] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:05:01.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:01 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:01] INFO grpc.py:275 (2): Received request to create bdev bdev_1d3b615a-3ba9-48ba-845e-70403d71658f from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:01.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:01 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:01] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 10 2024-09-02T12:05:01.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:01 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:01.420111] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f rbd disk to lun 2024-09-02T12:05:01.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:01 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:01] INFO grpc.py:770 (2): Received request to add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:05:01.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:01 smithi087 ceph-mon[21100]: pgmap v342: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 1.8 KiB/s wr, 72 op/s 2024-09-02T12:05:01.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:01 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:01] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:05:01.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:01 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:01] INFO grpc.py:275 (2): Received request to create bdev bdev_1d3b615a-3ba9-48ba-845e-70403d71658f from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:01.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:01 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:01] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 10 2024-09-02T12:05:01.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:01 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:01.417194] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f rbd disk to lun 2024-09-02T12:05:01.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:01 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:01] INFO grpc.py:770 (2): Received request to add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:05:01.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-mon[17163]: pgmap v342: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 1.8 KiB/s wr, 72 op/s 2024-09-02T12:05:01.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:01] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 3, context: , client address: IPv4 172.21.15.38:49812 2024-09-02T12:05:01.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:01] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:01.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:01] INFO grpc.py:275 (2): Received request to create bdev bdev_1d3b615a-3ba9-48ba-845e-70403d71658f from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:49812 2024-09-02T12:05:01.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:01] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 10 2024-09-02T12:05:01.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:01.391129] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f rbd disk to lun 2024-09-02T12:05:01.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:01 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:01] INFO grpc.py:770 (2): Received request to add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3, context: , client address: IPv4 172.21.15.38:49812 2024-09-02T12:05:01.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:01 smithi094 ceph-mon[20404]: pgmap v342: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 1.8 KiB/s wr, 72 op/s 2024-09-02T12:05:01.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:01] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:05:01.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:01] INFO grpc.py:275 (2): Received request to create bdev bdev_1d3b615a-3ba9-48ba-845e-70403d71658f from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:01.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:01] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 10 2024-09-02T12:05:01.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:01.418827] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f rbd disk to lun 2024-09-02T12:05:01.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:01] INFO grpc.py:770 (2): Received request to add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:05:02.288 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:02.288 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:02.288 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage39 2024-09-02T12:05:02.288 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage39 --load-balancing-group 4 2024-09-02T12:05:02.366 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:02Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:03.131 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 19 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:05:03.419 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:03] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 4, context: , client address: IPv4 172.21.15.38:52358 2024-09-02T12:05:03.420 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:03] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:03.420 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:03] INFO grpc.py:275 (2): Received request to create bdev bdev_755c5504-369c-43fc-9da2-ccd5a6361150 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:52358 2024-09-02T12:05:03.420 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 10 2024-09-02T12:05:03.420 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:03.115427] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 rbd disk to lun 2024-09-02T12:05:03.420 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:03] INFO grpc.py:770 (2): Received request to add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4, context: , client address: IPv4 172.21.15.38:52358 2024-09-02T12:05:03.420 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:03] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:03] INFO grpc.py:275 (2): Received request to create bdev bdev_755c5504-369c-43fc-9da2-ccd5a6361150 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 10 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:03.143867] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 rbd disk to lun 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:03 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:03] INFO grpc.py:770 (2): Received request to add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:03] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:03] INFO grpc.py:275 (2): Received request to create bdev bdev_755c5504-369c-43fc-9da2-ccd5a6361150 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 10 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:03.146637] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 rbd disk to lun 2024-09-02T12:05:03.421 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:03] INFO grpc.py:770 (2): Received request to add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:05:03.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:03] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:05:03.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:03] INFO grpc.py:275 (2): Received request to create bdev bdev_755c5504-369c-43fc-9da2-ccd5a6361150 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:03.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 10 2024-09-02T12:05:03.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:03.143218] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 rbd disk to lun 2024-09-02T12:05:03.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:03 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:03] INFO grpc.py:770 (2): Received request to add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:05:03.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:03 smithi087 ceph-mon[21100]: pgmap v343: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.4 KiB/s wr, 74 op/s 2024-09-02T12:05:03.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:03 smithi064 ceph-mon[17163]: pgmap v343: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.4 KiB/s wr, 74 op/s 2024-09-02T12:05:03.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:03 smithi094 ceph-mon[20404]: pgmap v343: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.4 KiB/s wr, 74 op/s 2024-09-02T12:05:03.962 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:03.962 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:03.962 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage40 2024-09-02T12:05:03.962 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode2 --rbd-pool mypool --rbd-image myimage40 --load-balancing-group 1 2024-09-02T12:05:04.037 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:04Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:04.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:04.752 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 20 to nqn.2016-06.io.spdk:cnode2: Successful 2024-09-02T12:05:04.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:04.853 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:04.884 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:04.956 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:05.017 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:04 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:04] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:05:05.017 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:04 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:04] INFO grpc.py:275 (2): Received request to create bdev bdev_0768e8b5-df41-497d-9343-ac44cee11a6a from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:05.017 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:04 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:04] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 10 2024-09-02T12:05:05.018 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:04 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:04.763366] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a rbd disk to lun 2024-09-02T12:05:05.018 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:04 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:04] INFO grpc.py:770 (2): Received request to add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:05:05.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:05.048 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:04] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:05:05.048 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:04] INFO grpc.py:275 (2): Received request to create bdev bdev_0768e8b5-df41-497d-9343-ac44cee11a6a from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:05.049 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:04] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 10 2024-09-02T12:05:05.049 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:04.765544] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a rbd disk to lun 2024-09-02T12:05:05.049 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:04] INFO grpc.py:770 (2): Received request to add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:05:05.060 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:05.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:04] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode2, ana group 1, context: , client address: IPv4 172.21.15.38:35438 2024-09-02T12:05:05.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:04] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:05.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:04] INFO grpc.py:275 (2): Received request to create bdev bdev_0768e8b5-df41-497d-9343-ac44cee11a6a from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:35438 2024-09-02T12:05:05.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:04] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 10 2024-09-02T12:05:05.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:04.735348] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a rbd disk to lun 2024-09-02T12:05:05.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:04] INFO grpc.py:770 (2): Received request to add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a to nqn.2016-06.io.spdk:cnode2 with ANA group id 1, context: , client address: IPv4 172.21.15.38:35438 2024-09-02T12:05:05.133 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:05.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:05.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:04 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:04] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:05:05.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:04 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:04] INFO grpc.py:275 (2): Received request to create bdev bdev_0768e8b5-df41-497d-9343-ac44cee11a6a from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:05.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:04 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:04] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 10 2024-09-02T12:05:05.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:04 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:04.761934] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a rbd disk to lun 2024-09-02T12:05:05.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:04 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:04] INFO grpc.py:770 (2): Received request to add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:05:05.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:05.267 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:05.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:05.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:05 smithi064 ceph-mon[17163]: pgmap v344: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.4 KiB/s wr, 72 op/s 2024-09-02T12:05:05.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:05 smithi094 ceph-mon[20404]: pgmap v344: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.4 KiB/s wr, 72 op/s 2024-09-02T12:05:05.630 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:05.630 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:05:05.630 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:05.631 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 20 2024-09-02T12:05:05.631 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:05.631 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage41 2024-09-02T12:05:05.632 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage41 --load-balancing-group 2 2024-09-02T12:05:05.712 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:05Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:05.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:05 smithi087 ceph-mon[21100]: pgmap v344: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 1.4 KiB/s wr, 72 op/s 2024-09-02T12:05:06.472 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 1 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:06.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:06] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:06.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:06] INFO grpc.py:275 (2): Received request to create bdev bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:06.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 11 2024-09-02T12:05:06.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:06.484371] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 rbd disk to lun 2024-09-02T12:05:06.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:06 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:06] INFO grpc.py:770 (2): Received request to add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:05:06.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:06] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, context: , client address: IPv4 172.21.15.38:40162 2024-09-02T12:05:06.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:06] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:06.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:06] INFO grpc.py:275 (2): Received request to create bdev bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:40162 2024-09-02T12:05:06.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 11 2024-09-02T12:05:06.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:06.454716] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 rbd disk to lun 2024-09-02T12:05:06.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:06 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:06] INFO grpc.py:770 (2): Received request to add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2, context: , client address: IPv4 172.21.15.38:40162 2024-09-02T12:05:06.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:06] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:06.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:06] INFO grpc.py:275 (2): Received request to create bdev bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:06.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 11 2024-09-02T12:05:06.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:06.481674] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 rbd disk to lun 2024-09-02T12:05:06.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:06] INFO grpc.py:770 (2): Received request to add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:05:06.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:06] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:06.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:06] INFO grpc.py:275 (2): Received request to create bdev bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:06.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:06] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 11 2024-09-02T12:05:06.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:06.483501] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 rbd disk to lun 2024-09-02T12:05:06.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:06 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:06] INFO grpc.py:770 (2): Received request to add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:05:07.333 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:07.333 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:07.333 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage42 2024-09-02T12:05:07.333 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage42 --load-balancing-group 3 2024-09-02T12:05:07.406 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:07Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:07.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:07 smithi087 ceph-mon[21100]: pgmap v345: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 1.8 KiB/s wr, 73 op/s 2024-09-02T12:05:07.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:07 smithi064 ceph-mon[17163]: pgmap v345: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 1.8 KiB/s wr, 73 op/s 2024-09-02T12:05:07.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:07 smithi094 ceph-mon[20404]: pgmap v345: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 1.8 KiB/s wr, 73 op/s 2024-09-02T12:05:08.180 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 2 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:08.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:08] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:08.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:08] INFO grpc.py:275 (2): Received request to create bdev bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:08.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 11 2024-09-02T12:05:08.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:08.190582] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 rbd disk to lun 2024-09-02T12:05:08.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:08 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:08] INFO grpc.py:770 (2): Received request to add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:05:08.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:08] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:08.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:08] INFO grpc.py:275 (2): Received request to create bdev bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:08.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 11 2024-09-02T12:05:08.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:08.189569] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 rbd disk to lun 2024-09-02T12:05:08.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:08 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:08] INFO grpc.py:770 (2): Received request to add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:05:08.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:08] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, context: , client address: IPv4 172.21.15.38:54810 2024-09-02T12:05:08.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:08] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:08.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:08] INFO grpc.py:275 (2): Received request to create bdev bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:54810 2024-09-02T12:05:08.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 11 2024-09-02T12:05:08.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:08.163737] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 rbd disk to lun 2024-09-02T12:05:08.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:08 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:08] INFO grpc.py:770 (2): Received request to add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3, context: , client address: IPv4 172.21.15.38:54810 2024-09-02T12:05:08.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:08] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:08.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:08] INFO grpc.py:275 (2): Received request to create bdev bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:08.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:08] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 11 2024-09-02T12:05:08.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:08.191923] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 rbd disk to lun 2024-09-02T12:05:08.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:08] INFO grpc.py:770 (2): Received request to add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:05:09.019 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:09.019 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:09.019 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage43 2024-09-02T12:05:09.019 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage43 --load-balancing-group 4 2024-09-02T12:05:09.112 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:09Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:09.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:09 smithi087 ceph-mon[21100]: pgmap v346: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:05:09.819 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-mon[17163]: pgmap v346: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:05:09.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:09 smithi094 ceph-mon[20404]: pgmap v346: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:05:09.856 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 3 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:10.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:09] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, context: , client address: IPv4 172.21.15.38:58378 2024-09-02T12:05:10.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:09] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:05:10.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:09] INFO grpc.py:275 (2): Received request to create bdev bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:58378 2024-09-02T12:05:10.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:09] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 11 2024-09-02T12:05:10.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:09.840034] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 rbd disk to lun 2024-09-02T12:05:10.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:09 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:09] INFO grpc.py:770 (2): Received request to add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4, context: , client address: IPv4 172.21.15.38:58378 2024-09-02T12:05:10.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:09 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:09] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:10.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:09 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:09] INFO grpc.py:275 (2): Received request to create bdev bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:10.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:09 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:09] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 11 2024-09-02T12:05:10.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:09 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:09.868256] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 rbd disk to lun 2024-09-02T12:05:10.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:09 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:09] INFO grpc.py:770 (2): Received request to add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:05:10.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:09 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:09] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:10.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:09 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:09] INFO grpc.py:275 (2): Received request to create bdev bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:10.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:09 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:09] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 11 2024-09-02T12:05:10.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:09 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:09.865317] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 rbd disk to lun 2024-09-02T12:05:10.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:09 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:09] INFO grpc.py:770 (2): Received request to add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:05:10.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:10.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:09] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:10.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:09] INFO grpc.py:275 (2): Received request to create bdev bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:10.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:09] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 11 2024-09-02T12:05:10.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:09.868739] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 rbd disk to lun 2024-09-02T12:05:10.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:09] INFO grpc.py:770 (2): Received request to add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:05:10.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:10.433 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:10.464 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:10.535 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:10.605 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:10.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:10.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:10.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:10.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage44 2024-09-02T12:05:10.695 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage44 --load-balancing-group 1 2024-09-02T12:05:10.699 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:10 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/3947371988' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:10.711 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:10.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:10 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/3947371988' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:10.777 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:10Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:10.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:10.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:10 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/3947371988' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:10.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:10.845 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:10.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:11.515 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 4 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:11.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:11 smithi087 ceph-mon[21100]: pgmap v347: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 156 KiB/s rd, 2.1 KiB/s wr, 84 op/s 2024-09-02T12:05:11.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:11 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:11] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:11.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:11 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:11] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e4936-9e03-425b-ba93-b65323d44402 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:11.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:11 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:11] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 11 2024-09-02T12:05:11.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:11 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:11.527171] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 rbd disk to lun 2024-09-02T12:05:11.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:11 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:11] INFO grpc.py:770 (2): Received request to add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:05:11.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-mon[17163]: pgmap v347: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 156 KiB/s rd, 2.1 KiB/s wr, 84 op/s 2024-09-02T12:05:11.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:11] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, context: , client address: IPv4 172.21.15.38:58476 2024-09-02T12:05:11.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:11] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:11.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:11] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e4936-9e03-425b-ba93-b65323d44402 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:58476 2024-09-02T12:05:11.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:11] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 11 2024-09-02T12:05:11.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:11.497066] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 rbd disk to lun 2024-09-02T12:05:11.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:11 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:11] INFO grpc.py:770 (2): Received request to add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1, context: , client address: IPv4 172.21.15.38:58476 2024-09-02T12:05:11.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:11 smithi094 ceph-mon[20404]: pgmap v347: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 156 KiB/s rd, 2.1 KiB/s wr, 84 op/s 2024-09-02T12:05:11.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:11] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:11.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:11] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e4936-9e03-425b-ba93-b65323d44402 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:11.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:11] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 11 2024-09-02T12:05:11.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:11.525637] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 rbd disk to lun 2024-09-02T12:05:11.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:11] INFO grpc.py:770 (2): Received request to add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:05:11.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:11 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:11] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:11.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:11 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:11] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e4936-9e03-425b-ba93-b65323d44402 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:11.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:11 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:11] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 11 2024-09-02T12:05:11.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:11 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:11.526588] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 rbd disk to lun 2024-09-02T12:05:11.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:11 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:11] INFO grpc.py:770 (2): Received request to add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:05:12.372 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:12.372 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:12.372 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage45 2024-09-02T12:05:12.372 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage45 --load-balancing-group 2 2024-09-02T12:05:12.463 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:12Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:13.229 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 5 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:13.452 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:13] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, context: , client address: IPv4 172.21.15.38:33842 2024-09-02T12:05:13.452 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:13] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:13.453 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:13] INFO grpc.py:275 (2): Received request to create bdev bdev_547cce68-47ad-4227-a432-c48eb15be142 from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:33842 2024-09-02T12:05:13.453 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 12 2024-09-02T12:05:13.453 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:13.213476] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_547cce68-47ad-4227-a432-c48eb15be142 rbd disk to lun 2024-09-02T12:05:13.453 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:13] INFO grpc.py:770 (2): Received request to add bdev_547cce68-47ad-4227-a432-c48eb15be142 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2, context: , client address: IPv4 172.21.15.38:33842 2024-09-02T12:05:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:13 smithi087 ceph-mon[21100]: pgmap v348: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:05:13.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:13] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:13.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:13] INFO grpc.py:275 (2): Received request to create bdev bdev_547cce68-47ad-4227-a432-c48eb15be142 from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:13.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 12 2024-09-02T12:05:13.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:13.239659] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_547cce68-47ad-4227-a432-c48eb15be142 rbd disk to lun 2024-09-02T12:05:13.501 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:13 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:13] INFO grpc.py:770 (2): Received request to add bdev_547cce68-47ad-4227-a432-c48eb15be142 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:05:13.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:13 smithi094 ceph-mon[20404]: pgmap v348: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:05:13.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:13] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:13.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:13] INFO grpc.py:275 (2): Received request to create bdev bdev_547cce68-47ad-4227-a432-c48eb15be142 from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:13.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 12 2024-09-02T12:05:13.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:13.238099] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_547cce68-47ad-4227-a432-c48eb15be142 rbd disk to lun 2024-09-02T12:05:13.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:13] INFO grpc.py:770 (2): Received request to add bdev_547cce68-47ad-4227-a432-c48eb15be142 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:05:13.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:13] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:13.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:13] INFO grpc.py:275 (2): Received request to create bdev bdev_547cce68-47ad-4227-a432-c48eb15be142 from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:13.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:13] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 12 2024-09-02T12:05:13.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:13.234658] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_547cce68-47ad-4227-a432-c48eb15be142 rbd disk to lun 2024-09-02T12:05:13.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:13 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:13] INFO grpc.py:770 (2): Received request to add bdev_547cce68-47ad-4227-a432-c48eb15be142 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:05:13.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:13 smithi064 ceph-mon[17163]: pgmap v348: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 124 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:05:14.099 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:14.099 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:14.099 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage46 2024-09-02T12:05:14.099 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage46 --load-balancing-group 3 2024-09-02T12:05:14.175 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:14.926 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 6 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:15.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:14 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:14] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:15.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:14 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:14] INFO grpc.py:275 (2): Received request to create bdev bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:15.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:14 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:14] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 12 2024-09-02T12:05:15.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:14 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:14.935676] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 rbd disk to lun 2024-09-02T12:05:15.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:14 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:14] INFO grpc.py:770 (2): Received request to add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:05:15.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:14 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:14] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:15.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:14 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:14] INFO grpc.py:275 (2): Received request to create bdev bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:15.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:14 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:14] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 12 2024-09-02T12:05:15.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:14 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:14.936860] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 rbd disk to lun 2024-09-02T12:05:15.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:14 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:14] INFO grpc.py:770 (2): Received request to add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:05:15.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:14 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:14] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, context: , client address: IPv4 172.21.15.38:41158 2024-09-02T12:05:15.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:14 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:14] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:15.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:14 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:14] INFO grpc.py:275 (2): Received request to create bdev bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:41158 2024-09-02T12:05:15.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:14 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:14] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 12 2024-09-02T12:05:15.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:14 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:14.910313] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 rbd disk to lun 2024-09-02T12:05:15.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:14 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:14] INFO grpc.py:770 (2): Received request to add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3, context: , client address: IPv4 172.21.15.38:41158 2024-09-02T12:05:15.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:14] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:15.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:14] INFO grpc.py:275 (2): Received request to create bdev bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:15.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:14] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 12 2024-09-02T12:05:15.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:14.939289] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 rbd disk to lun 2024-09-02T12:05:15.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:14] INFO grpc.py:770 (2): Received request to add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:05:15.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:15 smithi087 ceph-mon[21100]: pgmap v349: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 1.5 KiB/s wr, 70 op/s 2024-09-02T12:05:15.791 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:15.791 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:15.791 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage47 2024-09-02T12:05:15.791 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage47 --load-balancing-group 4 2024-09-02T12:05:15.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:15 smithi064 ceph-mon[17163]: pgmap v349: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 1.5 KiB/s wr, 70 op/s 2024-09-02T12:05:15.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:15 smithi094 ceph-mon[20404]: pgmap v349: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 1.5 KiB/s wr, 70 op/s 2024-09-02T12:05:15.865 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:15Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:15.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:15.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:16.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:16.045 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:16.115 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:16.187 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:16.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:16.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:16.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:16.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:16.427 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:16.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:16.626 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 7 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:16.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:16] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:16.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:16] INFO grpc.py:275 (2): Received request to create bdev bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:16.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:16] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 12 2024-09-02T12:05:16.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:16.638213] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 rbd disk to lun 2024-09-02T12:05:16.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:16] INFO grpc.py:770 (2): Received request to add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:05:16.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:16 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:16] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, context: , client address: IPv4 172.21.15.38:59940 2024-09-02T12:05:16.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:16 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:16] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:16.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:16 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:16] INFO grpc.py:275 (2): Received request to create bdev bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:59940 2024-09-02T12:05:16.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:16 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:16] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 12 2024-09-02T12:05:16.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:16 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:16.610344] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 rbd disk to lun 2024-09-02T12:05:16.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:16 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:16] INFO grpc.py:770 (2): Received request to add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4, context: , client address: IPv4 172.21.15.38:59940 2024-09-02T12:05:16.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:16] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:16.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:16] INFO grpc.py:275 (2): Received request to create bdev bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:16.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:16] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 12 2024-09-02T12:05:16.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:16.636586] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 rbd disk to lun 2024-09-02T12:05:16.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:16] INFO grpc.py:770 (2): Received request to add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:05:16.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:16 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:16] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:16.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:16 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:16] INFO grpc.py:275 (2): Received request to create bdev bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:16.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:16 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:16] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 12 2024-09-02T12:05:16.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:16 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:16.637628] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 rbd disk to lun 2024-09-02T12:05:16.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:16 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:16] INFO grpc.py:770 (2): Received request to add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:05:17.500 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:17.500 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:17.500 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage48 2024-09-02T12:05:17.500 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage48 --load-balancing-group 1 2024-09-02T12:05:17.580 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:17Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:17 smithi087 ceph-mon[21100]: pgmap v350: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 2.0 KiB/s wr, 79 op/s 2024-09-02T12:05:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:17 smithi064 ceph-mon[17163]: pgmap v350: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 2.0 KiB/s wr, 79 op/s 2024-09-02T12:05:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:17 smithi094 ceph-mon[20404]: pgmap v350: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 2.0 KiB/s wr, 79 op/s 2024-09-02T12:05:18.336 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 8 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:18.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:18] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, context: , client address: IPv4 172.21.15.38:46862 2024-09-02T12:05:18.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:18] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:18.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:18] INFO grpc.py:275 (2): Received request to create bdev bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:46862 2024-09-02T12:05:18.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 12 2024-09-02T12:05:18.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:18.318615] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e rbd disk to lun 2024-09-02T12:05:18.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:18 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:18] INFO grpc.py:770 (2): Received request to add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e to nqn.2016-06.io.spdk:cnode3 with ANA group id 1, context: , client address: IPv4 172.21.15.38:46862 2024-09-02T12:05:18.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:18] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:18.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:18] INFO grpc.py:275 (2): Received request to create bdev bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:18.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 12 2024-09-02T12:05:18.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:18.350032] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e rbd disk to lun 2024-09-02T12:05:18.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:18] INFO grpc.py:770 (2): Received request to add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:05:18.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:18] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:18.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:18] INFO grpc.py:275 (2): Received request to create bdev bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:18.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 12 2024-09-02T12:05:18.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:18.346935] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e rbd disk to lun 2024-09-02T12:05:18.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:18 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:18] INFO grpc.py:770 (2): Received request to add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:05:18.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:18] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:18.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:18] INFO grpc.py:275 (2): Received request to create bdev bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:18.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:18] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 12 2024-09-02T12:05:18.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:18.345566] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e rbd disk to lun 2024-09-02T12:05:18.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:18 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:18] INFO grpc.py:770 (2): Received request to add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:05:19.211 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:19.211 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:19.211 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage49 2024-09-02T12:05:19.211 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage49 --load-balancing-group 2 2024-09-02T12:05:19.301 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:19.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:19 smithi087 ceph-mon[21100]: pgmap v351: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:05:19.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:19 smithi064 ceph-mon[17163]: pgmap v351: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:05:19.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:19 smithi094 ceph-mon[20404]: pgmap v351: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:05:20.036 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 9 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:20.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:19 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:19] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, context: , client address: IPv4 172.21.15.38:45364 2024-09-02T12:05:20.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:20] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:05:20.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:20] INFO grpc.py:275 (2): Received request to create bdev bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:45364 2024-09-02T12:05:20.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 13 2024-09-02T12:05:20.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:20.018042] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f rbd disk to lun 2024-09-02T12:05:20.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:20 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:20] INFO grpc.py:770 (2): Received request to add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f to nqn.2016-06.io.spdk:cnode3 with ANA group id 2, context: , client address: IPv4 172.21.15.38:45364 2024-09-02T12:05:20.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:20] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:20.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:20] INFO grpc.py:275 (2): Received request to create bdev bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:20.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 13 2024-09-02T12:05:20.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:20.048605] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f rbd disk to lun 2024-09-02T12:05:20.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:20] INFO grpc.py:770 (2): Received request to add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:05:20.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:20] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:20.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:20] INFO grpc.py:275 (2): Received request to create bdev bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:20.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 13 2024-09-02T12:05:20.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:20.050231] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f rbd disk to lun 2024-09-02T12:05:20.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:20 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:20] INFO grpc.py:770 (2): Received request to add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:05:20.472 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:20] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:20.472 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:20] INFO grpc.py:275 (2): Received request to create bdev bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:20.472 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:20] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 13 2024-09-02T12:05:20.472 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:20.047725] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f rbd disk to lun 2024-09-02T12:05:20.472 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:20 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:20] INFO grpc.py:770 (2): Received request to add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:05:20.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:20 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/4212097307' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:20.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:20 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/4212097307' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:20.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:20 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/4212097307' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:20.869 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:20.869 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:20.869 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage50 2024-09-02T12:05:20.869 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage50 --load-balancing-group 3 2024-09-02T12:05:20.968 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:20Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:21.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:21.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:21.592 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:21.622 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:21.694 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:21.730 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-mon[17163]: pgmap v352: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2024-09-02T12:05:21.730 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:21] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, context: , client address: IPv4 172.21.15.38:49500 2024-09-02T12:05:21.730 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:21] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:21.730 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:21] INFO grpc.py:275 (2): Received request to create bdev bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:49500 2024-09-02T12:05:21.730 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:21] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 13 2024-09-02T12:05:21.730 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:21.729560] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef rbd disk to lun 2024-09-02T12:05:21.748 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:21 smithi087 ceph-mon[21100]: pgmap v352: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2024-09-02T12:05:21.749 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:21 smithi094 ceph-mon[20404]: pgmap v352: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 1.9 KiB/s wr, 68 op/s 2024-09-02T12:05:21.749 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 10 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:21.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:21.799 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:21.872 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:21.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:21.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:21.999 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:21 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:21] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:21.999 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:21 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:21] INFO grpc.py:275 (2): Received request to create bdev bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:21.999 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:21 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:21] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 13 2024-09-02T12:05:21.999 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:21 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:21.756131] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef rbd disk to lun 2024-09-02T12:05:21.999 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:21 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:21] INFO grpc.py:770 (2): Received request to add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:05:22.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:22.028 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:21] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:22.028 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:21] INFO grpc.py:275 (2): Received request to create bdev bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:22.028 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:21] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 13 2024-09-02T12:05:22.028 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:21.758229] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef rbd disk to lun 2024-09-02T12:05:22.029 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:21] INFO grpc.py:770 (2): Received request to add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:05:22.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:22.061 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:21] INFO grpc.py:770 (2): Received request to add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef to nqn.2016-06.io.spdk:cnode3 with ANA group id 3, context: , client address: IPv4 172.21.15.38:49500 2024-09-02T12:05:22.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:21 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:21] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:22.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:21 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:21] INFO grpc.py:275 (2): Received request to create bdev bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:22.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:21 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:21] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 13 2024-09-02T12:05:22.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:21 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:21.761053] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef rbd disk to lun 2024-09-02T12:05:22.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:21 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:21] INFO grpc.py:770 (2): Received request to add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:05:22.597 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:22.597 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:22.597 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage51 2024-09-02T12:05:22.597 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage51 --load-balancing-group 4 2024-09-02T12:05:22.671 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:22Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:23.423 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 11 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:23.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:23] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:23.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:23] INFO grpc.py:275 (2): Received request to create bdev bdev_40f6a294-315b-40eb-9839-236059ddbec0 from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:23.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 13 2024-09-02T12:05:23.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:23.436017] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_40f6a294-315b-40eb-9839-236059ddbec0 rbd disk to lun 2024-09-02T12:05:23.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:23 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:23] INFO grpc.py:770 (2): Received request to add bdev_40f6a294-315b-40eb-9839-236059ddbec0 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:05:23.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:23 smithi087 ceph-mon[21100]: pgmap v353: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:05:23.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:23] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:23.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:23] INFO grpc.py:275 (2): Received request to create bdev bdev_40f6a294-315b-40eb-9839-236059ddbec0 from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:23.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 13 2024-09-02T12:05:23.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:23.432412] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_40f6a294-315b-40eb-9839-236059ddbec0 rbd disk to lun 2024-09-02T12:05:23.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:23 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:23] INFO grpc.py:770 (2): Received request to add bdev_40f6a294-315b-40eb-9839-236059ddbec0 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:05:23.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-mon[17163]: pgmap v353: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:05:23.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:23] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, context: , client address: IPv4 172.21.15.38:60534 2024-09-02T12:05:23.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:23] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:23.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:23] INFO grpc.py:275 (2): Received request to create bdev bdev_40f6a294-315b-40eb-9839-236059ddbec0 from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:60534 2024-09-02T12:05:23.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 13 2024-09-02T12:05:23.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:23.407935] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_40f6a294-315b-40eb-9839-236059ddbec0 rbd disk to lun 2024-09-02T12:05:23.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:23] INFO grpc.py:770 (2): Received request to add bdev_40f6a294-315b-40eb-9839-236059ddbec0 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4, context: , client address: IPv4 172.21.15.38:60534 2024-09-02T12:05:23.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:23 smithi094 ceph-mon[20404]: pgmap v353: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:05:23.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:23] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:23.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:23] INFO grpc.py:275 (2): Received request to create bdev bdev_40f6a294-315b-40eb-9839-236059ddbec0 from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:23.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:23] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 13 2024-09-02T12:05:23.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:23.435056] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_40f6a294-315b-40eb-9839-236059ddbec0 rbd disk to lun 2024-09-02T12:05:23.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:23 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:23] INFO grpc.py:770 (2): Received request to add bdev_40f6a294-315b-40eb-9839-236059ddbec0 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:05:24.292 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:24.292 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:24.292 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage52 2024-09-02T12:05:24.292 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage52 --load-balancing-group 1 2024-09-02T12:05:24.381 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:24Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:25.126 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 12 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:25.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:25] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:25.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:25] INFO grpc.py:275 (2): Received request to create bdev bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:25.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 13 2024-09-02T12:05:25.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:25.139334] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f rbd disk to lun 2024-09-02T12:05:25.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:25 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:25] INFO grpc.py:770 (2): Received request to add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:05:25.488 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:25] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, context: , client address: IPv4 172.21.15.38:44894 2024-09-02T12:05:25.488 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:25] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:25] INFO grpc.py:275 (2): Received request to create bdev bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:44894 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 13 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:25.108469] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f rbd disk to lun 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:25] INFO grpc.py:770 (2): Received request to add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f to nqn.2016-06.io.spdk:cnode3 with ANA group id 1, context: , client address: IPv4 172.21.15.38:44894 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:25] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:25] INFO grpc.py:275 (2): Received request to create bdev bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 13 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:25.134733] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f rbd disk to lun 2024-09-02T12:05:25.489 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:25 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:25] INFO grpc.py:770 (2): Received request to add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:05:25.491 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:25] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:25.491 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:25] INFO grpc.py:275 (2): Received request to create bdev bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:25.491 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:25] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 13 2024-09-02T12:05:25.491 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:25.137706] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f rbd disk to lun 2024-09-02T12:05:25.491 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:25 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:25] INFO grpc.py:770 (2): Received request to add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:05:25.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:25 smithi087 ceph-mon[21100]: pgmap v354: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 1.7 KiB/s wr, 93 op/s 2024-09-02T12:05:25.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:25 smithi064 ceph-mon[17163]: pgmap v354: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 1.7 KiB/s wr, 93 op/s 2024-09-02T12:05:25.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:25 smithi094 ceph-mon[20404]: pgmap v354: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 1.7 KiB/s wr, 93 op/s 2024-09-02T12:05:25.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:25.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:25.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage53 2024-09-02T12:05:25.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage53 --load-balancing-group 2 2024-09-02T12:05:26.037 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:26Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:26.854 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 13 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:27.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:27.093 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:26] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, context: , client address: IPv4 172.21.15.38:36076 2024-09-02T12:05:27.093 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:26] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:27.093 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:26] INFO grpc.py:275 (2): Received request to create bdev bdev_b5597828-c750-4702-aa14-9709fb7d1d91 from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:36076 2024-09-02T12:05:27.093 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:26] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 14 2024-09-02T12:05:27.093 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:26.837901] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 rbd disk to lun 2024-09-02T12:05:27.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:26 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:26] INFO grpc.py:770 (2): Received request to add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2, context: , client address: IPv4 172.21.15.38:36076 2024-09-02T12:05:27.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:27.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:27.196 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:26] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:27.196 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:26] INFO grpc.py:275 (2): Received request to create bdev bdev_b5597828-c750-4702-aa14-9709fb7d1d91 from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:27.196 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:26] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 14 2024-09-02T12:05:27.196 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:26.863983] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 rbd disk to lun 2024-09-02T12:05:27.196 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:26 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:26] INFO grpc.py:770 (2): Received request to add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:05:27.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:27.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:26] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:27.208 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:26] INFO grpc.py:275 (2): Received request to create bdev bdev_b5597828-c750-4702-aa14-9709fb7d1d91 from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:27.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:26] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 14 2024-09-02T12:05:27.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:26.865535] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 rbd disk to lun 2024-09-02T12:05:27.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:26 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:26] INFO grpc.py:770 (2): Received request to add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:05:27.278 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:27.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:26] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:27.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:26] INFO grpc.py:275 (2): Received request to create bdev bdev_b5597828-c750-4702-aa14-9709fb7d1d91 from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:27.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:26] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 14 2024-09-02T12:05:27.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:26.863083] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 rbd disk to lun 2024-09-02T12:05:27.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:26 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:26] INFO grpc.py:770 (2): Received request to add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:05:27.348 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:27.381 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:27.454 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:27.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:27.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:27.588 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:27.620 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:27 smithi094 ceph-mon[20404]: pgmap v355: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:05:27.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:27.728 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:27.728 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:27.728 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage54 2024-09-02T12:05:27.728 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage54 --load-balancing-group 3 2024-09-02T12:05:27.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:27 smithi087 ceph-mon[21100]: pgmap v355: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:05:27.802 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:27Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:27.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:27 smithi064 ceph-mon[17163]: pgmap v355: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:05:28.532 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 14 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:28.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:28] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, context: , client address: IPv4 172.21.15.38:52540 2024-09-02T12:05:28.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:28] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:28.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:28] INFO grpc.py:275 (2): Received request to create bdev bdev_70de1546-c18d-49bb-b824-32b6b807a5ab from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:52540 2024-09-02T12:05:28.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 14 2024-09-02T12:05:28.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:28.514198] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab rbd disk to lun 2024-09-02T12:05:28.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:28] INFO grpc.py:770 (2): Received request to add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab to nqn.2016-06.io.spdk:cnode3 with ANA group id 3, context: , client address: IPv4 172.21.15.38:52540 2024-09-02T12:05:28.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:28] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:28.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:28] INFO grpc.py:275 (2): Received request to create bdev bdev_70de1546-c18d-49bb-b824-32b6b807a5ab from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:28.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 14 2024-09-02T12:05:28.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:28.545384] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab rbd disk to lun 2024-09-02T12:05:28.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:28 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:28] INFO grpc.py:770 (2): Received request to add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:05:28.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:28] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:28.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:28] INFO grpc.py:275 (2): Received request to create bdev bdev_70de1546-c18d-49bb-b824-32b6b807a5ab from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:28.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 14 2024-09-02T12:05:28.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:28.543337] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab rbd disk to lun 2024-09-02T12:05:28.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:28 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:28] INFO grpc.py:770 (2): Received request to add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:05:29.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:28] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:29.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:28] INFO grpc.py:275 (2): Received request to create bdev bdev_70de1546-c18d-49bb-b824-32b6b807a5ab from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:29.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:28] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 14 2024-09-02T12:05:29.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:28.546092] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab rbd disk to lun 2024-09-02T12:05:29.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:28 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:28] INFO grpc.py:770 (2): Received request to add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:05:29.328 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:29.328 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:29.329 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage55 2024-09-02T12:05:29.329 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage55 --load-balancing-group 4 2024-09-02T12:05:29.407 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:29Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:29.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:29 smithi087 ceph-mon[21100]: pgmap v356: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:05:29.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:29 smithi064 ceph-mon[17163]: pgmap v356: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:05:29.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:29 smithi094 ceph-mon[20404]: pgmap v356: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 1.7 KiB/s wr, 94 op/s 2024-09-02T12:05:30.132 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 15 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:30.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:30] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:30.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:30] INFO grpc.py:275 (2): Received request to create bdev bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:30.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 14 2024-09-02T12:05:30.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:30.136021] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 rbd disk to lun 2024-09-02T12:05:30.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:30 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:30] INFO grpc.py:770 (2): Received request to add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:05:30.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:30] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:30.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:30] INFO grpc.py:275 (2): Received request to create bdev bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:30.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 14 2024-09-02T12:05:30.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:30.142627] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 rbd disk to lun 2024-09-02T12:05:30.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:30] INFO grpc.py:770 (2): Received request to add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:05:30.501 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:30] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:30.501 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:30] INFO grpc.py:275 (2): Received request to create bdev bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:30.501 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 14 2024-09-02T12:05:30.501 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:30.140905] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 rbd disk to lun 2024-09-02T12:05:30.501 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:30] INFO grpc.py:770 (2): Received request to add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:05:30.502 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:30] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, context: , client address: IPv4 172.21.15.38:43812 2024-09-02T12:05:30.502 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:30] INFO cephutils.py:58 (2): ANA group list: [1, 2, 3, 4] 2024-09-02T12:05:30.502 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:30] INFO grpc.py:275 (2): Received request to create bdev bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:43812 2024-09-02T12:05:30.502 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:30] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 14 2024-09-02T12:05:30.502 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:30.114113] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 rbd disk to lun 2024-09-02T12:05:30.502 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:30] INFO grpc.py:770 (2): Received request to add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4, context: , client address: IPv4 172.21.15.38:43812 2024-09-02T12:05:30.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-mon[17163]: from='client.? 172.21.15.64:0/2349480510' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:30.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:05:30.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:30.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:30 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:30.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-mon[20404]: from='client.? 172.21.15.64:0/2349480510' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:30.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:05:30.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:30.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:30 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-mon[21100]: from='client.? 172.21.15.64:0/2349480510' entity='client.nvmeof.nvmeof.a' cmd={"prefix":"nvme-gw show", "pool":"mypool", "group":"mygroup0"} : dispatch 2024-09-02T12:05:31.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:05:31.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:30 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.097 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:31.097 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:31.097 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage56 2024-09-02T12:05:31.097 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage56 --load-balancing-group 1 2024-09-02T12:05:31.189 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:31Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:31.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-mon[17163]: pgmap v357: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 179 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:05:31.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:31 smithi094 ceph-mon[20404]: pgmap v357: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 179 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:05:31.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:31 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.997 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.997 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.997 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-mon[21100]: pgmap v357: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 179 KiB/s rd, 2.1 KiB/s wr, 85 op/s 2024-09-02T12:05:31.997 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:31.997 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:32.000 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 16 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:32.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:31] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:32.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:31] INFO grpc.py:275 (2): Received request to create bdev bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:32.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:31 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:31] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 14 2024-09-02T12:05:32.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:32.007183] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 rbd disk to lun 2024-09-02T12:05:32.250 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:32 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:32] INFO grpc.py:770 (2): Received request to add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:05:32.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:31] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, context: , client address: IPv4 172.21.15.38:32950 2024-09-02T12:05:32.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:31] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:32.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:31] INFO grpc.py:275 (2): Received request to create bdev bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:32950 2024-09-02T12:05:32.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:31] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 14 2024-09-02T12:05:32.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:31.983018] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 rbd disk to lun 2024-09-02T12:05:32.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:31 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:31] INFO grpc.py:770 (2): Received request to add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1, context: , client address: IPv4 172.21.15.38:32950 2024-09-02T12:05:32.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:31 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:31] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:32.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:31] INFO grpc.py:275 (2): Received request to create bdev bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:32.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:32] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 14 2024-09-02T12:05:32.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:32.011485] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 rbd disk to lun 2024-09-02T12:05:32.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:32] INFO grpc.py:770 (2): Received request to add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:05:32.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:31 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:31] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:32.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:31 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:31] INFO grpc.py:275 (2): Received request to create bdev bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:32.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:31 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:31] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 14 2024-09-02T12:05:32.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:32.010837] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 rbd disk to lun 2024-09-02T12:05:32.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:32 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:32] INFO grpc.py:770 (2): Received request to add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:05:32.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:32.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:32.753 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:32.774 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:32 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"} : dispatch 2024-09-02T12:05:32.775 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:32 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:05:32.775 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:32 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:05:32.775 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:32 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:32.785 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:32.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:32 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"} : dispatch 2024-09-02T12:05:32.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:32 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:05:32.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:32 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:05:32.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:32 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:32.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"} : dispatch 2024-09-02T12:05:32.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:05:32.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:05:32.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:32 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:05:32.856 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:32.865 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:32.865 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:32.866 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage57 2024-09-02T12:05:32.866 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage57 --load-balancing-group 2 2024-09-02T12:05:32.928 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:32.947 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:32Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:32.960 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:33.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:33.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:33.137 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:33.169 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:33.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:33.691 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 17 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:33.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:33 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:33] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:33.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:33 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:33] INFO grpc.py:275 (2): Received request to create bdev bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:33.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:33 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:33] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 15 2024-09-02T12:05:33.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:33 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:33.700998] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 rbd disk to lun 2024-09-02T12:05:33.959 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:33 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:33] INFO grpc.py:770 (2): Received request to add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:05:34.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:33 smithi087 ceph-mon[21100]: pgmap v358: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:05:34.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:33 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:33] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:34.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:33 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:33] INFO grpc.py:275 (2): Received request to create bdev bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:34.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:33 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:33] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 15 2024-09-02T12:05:34.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:33 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:33.701877] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 rbd disk to lun 2024-09-02T12:05:34.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:33 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:33] INFO grpc.py:770 (2): Received request to add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:05:34.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-mon[17163]: pgmap v358: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:05:34.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:33] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 2, context: , client address: IPv4 172.21.15.38:39924 2024-09-02T12:05:34.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:33] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:34.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:33] INFO grpc.py:275 (2): Received request to create bdev bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:39924 2024-09-02T12:05:34.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:33] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 15 2024-09-02T12:05:34.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:33.675615] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 rbd disk to lun 2024-09-02T12:05:34.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:33 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:33] INFO grpc.py:770 (2): Received request to add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2, context: , client address: IPv4 172.21.15.38:39924 2024-09-02T12:05:34.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:33 smithi094 ceph-mon[20404]: pgmap v358: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 1.7 KiB/s wr, 86 op/s 2024-09-02T12:05:34.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:33 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:33] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:05:34.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:33 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:33] INFO grpc.py:275 (2): Received request to create bdev bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:34.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:33 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:33] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 15 2024-09-02T12:05:34.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:33 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:33.700052] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 rbd disk to lun 2024-09-02T12:05:34.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:33 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:33] INFO grpc.py:770 (2): Received request to add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:05:34.557 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:34.557 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:34.558 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage58 2024-09-02T12:05:34.558 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage58 --load-balancing-group 3 2024-09-02T12:05:34.643 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:34Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:35.406 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 18 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:35.655 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:35] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 3, context: , client address: IPv4 172.21.15.38:37798 2024-09-02T12:05:35.655 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:35] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:35.656 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:35] INFO grpc.py:275 (2): Received request to create bdev bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:37798 2024-09-02T12:05:35.656 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 15 2024-09-02T12:05:35.656 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:35.387974] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 rbd disk to lun 2024-09-02T12:05:35.656 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:35] INFO grpc.py:770 (2): Received request to add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3, context: , client address: IPv4 172.21.15.38:37798 2024-09-02T12:05:35.708 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:35] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:35.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:35] INFO grpc.py:275 (2): Received request to create bdev bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:35.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 15 2024-09-02T12:05:35.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:35.418971] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 rbd disk to lun 2024-09-02T12:05:35.709 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:35 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:35] INFO grpc.py:770 (2): Received request to add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:05:35.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:35 smithi087 ceph-mon[21100]: pgmap v359: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:05:35.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:35] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:35.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:35] INFO grpc.py:275 (2): Received request to create bdev bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:35.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 15 2024-09-02T12:05:35.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:35.413750] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 rbd disk to lun 2024-09-02T12:05:35.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:35 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:35] INFO grpc.py:770 (2): Received request to add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:05:35.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:35 smithi094 ceph-mon[20404]: pgmap v359: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:05:35.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:35] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:05:35.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:35] INFO grpc.py:275 (2): Received request to create bdev bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:35.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:35] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 15 2024-09-02T12:05:35.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:35.416673] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 rbd disk to lun 2024-09-02T12:05:35.846 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:35 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:35] INFO grpc.py:770 (2): Received request to add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:05:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:35 smithi064 ceph-mon[17163]: pgmap v359: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 138 KiB/s rd, 1.6 KiB/s wr, 79 op/s 2024-09-02T12:05:36.275 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:36.275 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:36.275 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage59 2024-09-02T12:05:36.275 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage59 --load-balancing-group 4 2024-09-02T12:05:36.361 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:36Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:37.128 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 19 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:37.458 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:37.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:37.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:05:37.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:37.135742] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:05:37.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:05:37.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:37.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:37.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:05:37.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:37.138836] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:05:37.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:05:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:37] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 4, context: , client address: IPv4 172.21.15.38:39708 2024-09-02T12:05:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:37] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:39708 2024-09-02T12:05:37.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:05:37.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:37.109351] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:05:37.595 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4, context: , client address: IPv4 172.21.15.38:39708 2024-09-02T12:05:37.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:37] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:05:37.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:37.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:05:37.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:37.138157] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:05:37.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:37 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:05:38.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-mon[21100]: pgmap v360: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 2.0 KiB/s wr, 78 op/s 2024-09-02T12:05:38.020 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:38.021 INFO:tasks.workunit.client.0.smithi038.stderr:+ for ns in $(seq 1 $NVMEOF_NAMESPACES_COUNT) 2024-09-02T12:05:38.021 INFO:tasks.workunit.client.0.smithi038.stderr:+ image=myimage60 2024-09-02T12:05:38.021 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 namespace add --subsystem nqn.2016-06.io.spdk:cnode3 --rbd-pool mypool --rbd-image myimage60 --load-balancing-group 1 2024-09-02T12:05:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:37 smithi064 ceph-mon[17163]: pgmap v360: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 2.0 KiB/s wr, 78 op/s 2024-09-02T12:05:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:37 smithi094 ceph-mon[20404]: pgmap v360: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 2.0 KiB/s wr, 78 op/s 2024-09-02T12:05:38.099 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:38Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:38.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:38.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:38.333 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:38.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:38.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:38.507 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:38.539 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:38.613 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:38.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:38.716 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:38.747 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:38.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:38.875 INFO:tasks.workunit.client.0.smithi038.stdout:Adding namespace 20 to nqn.2016-06.io.spdk:cnode3: Successful 2024-09-02T12:05:39.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:39.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:39.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:05:39.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:38.887886] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:05:39.000 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:05:39.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace to nqn.2016-06.io.spdk:cnode3, ana group 1, context: , client address: IPv4 172.21.15.38:58748 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:38] INFO cephutils.py:38 (2): Caching response of the monitor: [1, 2, 3, 4] 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=, client address: IPv4 172.21.15.38:58748 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:05:38.858512] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:38 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1, context: , client address: IPv4 172.21.15.38:58748 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:38 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:39.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:38 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:39.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:38 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:05:39.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:38 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [2024-09-02 12:05:38.886699] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:05:39.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:38 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:05:39.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:05:39.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:05:39.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:05:39.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:38.885553] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:05:39.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:05:39.713 INFO:tasks.workunit.client.0.smithi038.stderr:+ (( image_index++ )) 2024-09-02T12:05:39.713 INFO:tasks.workunit.client.0.smithi038.stderr:+ list_subsystems 2024-09-02T12:05:39.714 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:05:39.714 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.64 2024-09-02T12:05:39.714 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 --format json subsystem list 2024-09-02T12:05:39.795 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:40.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:39 smithi087 ceph-mon[21100]: pgmap v361: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:05:40.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:39 smithi064 ceph-mon[17163]: pgmap v361: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:05:40.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:39 smithi094 ceph-mon[20404]: pgmap v361: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 1.7 KiB/s wr, 87 op/s 2024-09-02T12:05:40.469 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 1, 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 2040, 2024-09-02T12:05:40.470 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 1, 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 2040, 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:40.471 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 1, 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 2040, 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:05:40.472 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:05:40.473 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:05:40.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:40 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:40] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:38828 2024-09-02T12:05:41.289 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:05:41.289 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.87 2024-09-02T12:05:41.289 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.87 --server-port 5500 --format json subsystem list 2024-09-02T12:05:41.397 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:41Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:42.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:41 smithi087 ceph-mon[21100]: pgmap v362: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 1.9 KiB/s wr, 67 op/s 2024-09-02T12:05:42.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:41 smithi064 ceph-mon[17163]: pgmap v362: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 1.9 KiB/s wr, 67 op/s 2024-09-02T12:05:42.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:41 smithi094 ceph-mon[20404]: pgmap v362: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 1.9 KiB/s wr, 67 op/s 2024-09-02T12:05:42.151 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:05:42.151 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:05:42.151 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:05:42.151 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 2041, 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 4080, 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:42.152 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 2041, 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 4080, 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:05:42.153 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 2041, 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 4080, 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:05:42.154 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:05:42.500 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:05:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:42] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:59450 2024-09-02T12:05:42.999 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:05:42.999 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.94 2024-09-02T12:05:42.999 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.94 --server-port 5500 --format json subsystem list 2024-09-02T12:05:43.077 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:43Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:43.802 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:05:43.802 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 4081, 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 6120, 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:43.803 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 4081, 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 6120, 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:05:43.804 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 4081, 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 6120, 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:05:43.805 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:05:43.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:43.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:43.912 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:43.933 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:43 smithi087 ceph-mon[21100]: pgmap v363: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:05:43.943 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:44.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:44.085 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:44.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:43 smithi064 ceph-mon[17163]: pgmap v363: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:05:44.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:43 smithi094 ceph-mon[20404]: pgmap v363: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:05:44.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:05:43 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:05:43] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:56316 2024-09-02T12:05:44.118 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:44.190 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:44.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:44.293 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:44.324 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:44.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:44.658 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in "${!gateway_ips[@]}" 2024-09-02T12:05:44.659 INFO:tasks.workunit.client.0.smithi038.stderr:+ ip=172.21.15.99 2024-09-02T12:05:44.659 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.99 --server-port 5500 --format json subsystem list 2024-09-02T12:05:44.739 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:44Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout:{ 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "error_message": "Success", 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "subsystems": [ 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph81601849504533", 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 6121, 2024-09-02T12:05:45.469 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 8160, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph61660518994858", 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 6121, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 8160, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:45.470 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: }, 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: { 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "nqn": "nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "enable_ha": true, 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "serial_number": "Ceph22728624249465", 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "model_number": "Ceph bdev Controller", 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "min_cntlid": 6121, 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "max_cntlid": 8160, 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "namespace_count": 20, 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "subtype": "NVMe", 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "max_namespaces": 256 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: } 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: ], 2024-09-02T12:05:45.471 INFO:tasks.workunit.client.0.smithi038.stdout: "status": 0 2024-09-02T12:05:45.472 INFO:tasks.workunit.client.0.smithi038.stdout:} 2024-09-02T12:05:45.958 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:05:45 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:45] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:35604 2024-09-02T12:05:46.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:45 smithi087 ceph-mon[21100]: pgmap v364: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:05:46.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:45 smithi064 ceph-mon[17163]: pgmap v364: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:05:46.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:45 smithi094 ceph-mon[20404]: pgmap v364: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 1.5 KiB/s wr, 68 op/s 2024-09-02T12:05:46.345 INFO:tasks.workunit.client.0.smithi038.stderr:++ seq 1 3 2024-09-02T12:05:46.346 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:05:46.346 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:46.346 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 --format plain namespace list --subsystem nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:46.419 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:46Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:47.300 INFO:tasks.workunit.client.0.smithi038.stdout:Namespaces in subsystem nqn.2016-06.io.spdk:cnode1: 2024-09-02T12:05:47.300 INFO:tasks.workunit.client.0.smithi038.stdout: NSID Bdev RBD RBD Image Block UUID Load R/W IOs R/W MBs Read MBs Write MBs 2024-09-02T12:05:47.300 INFO:tasks.workunit.client.0.smithi038.stdout: Name Pool Image Size Size Balancing per per per per 2024-09-02T12:05:47.300 INFO:tasks.workunit.client.0.smithi038.stdout: Group second second second second 2024-09-02T12:05:47.300 INFO:tasks.workunit.client.0.smithi038.stdout: 1 bdev_6d3e8754-98d1- mypool myimage1 8 GiB 512 Bytes 6d3e8754-98d1-4474- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.300 INFO:tasks.workunit.client.0.smithi038.stdout: 4474-a559-ac49989ff693 a559-ac49989ff693 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 2 bdev_6ffe1b68-223c- mypool myimage2 8 GiB 512 Bytes 6ffe1b68-223c-46c9- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 46c9-a396-3a904e32f1f4 a396-3a904e32f1f4 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 3 bdev_f96044ec-462e- mypool myimage3 8 GiB 512 Bytes f96044ec-462e-4f71- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 4f71-a581-9eab30b51024 a581-9eab30b51024 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 4 bdev_036643fa-ace3- mypool myimage4 8 GiB 512 Bytes 036643fa-ace3-4623- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 4623-acfe-a405d326e7c7 acfe-a405d326e7c7 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 5 bdev_127a264a-a286- mypool myimage5 8 GiB 512 Bytes 127a264a-a286-4c07- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 4c07-95dc-3c381465fe2e 95dc-3c381465fe2e 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 6 bdev_c390a368-6527- mypool myimage6 8 GiB 512 Bytes c390a368-6527-4721- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 4721-81b5-0395d5ac0ff7 81b5-0395d5ac0ff7 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 7 bdev_55652567-0cc3- mypool myimage7 8 GiB 512 Bytes 55652567-0cc3-463c- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 463c-b02c-3c0bbfd57e1b b02c-3c0bbfd57e1b 2024-09-02T12:05:47.301 INFO:tasks.workunit.client.0.smithi038.stdout: 8 bdev_2e0e2ca4-1342- mypool myimage8 8 GiB 512 Bytes 2e0e2ca4-1342-437a- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 437a-9dd6-8744dde7e146 9dd6-8744dde7e146 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 9 bdev_202aeb00-7d9d- mypool myimage9 8 GiB 512 Bytes 202aeb00-7d9d-44a8- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 44a8-8d9e-190d60b8bd14 8d9e-190d60b8bd14 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 10 bdev_7c1f1adf-a4f7- mypool myimage10 8 GiB 512 Bytes 7c1f1adf-a4f7-48b5- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 48b5-a3ea-3233f3291978 a3ea-3233f3291978 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 11 bdev_6619312d-add9- mypool myimage11 8 GiB 512 Bytes 6619312d-add9-4d30- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 4d30-bd8d-c768d6f81dbc bd8d-c768d6f81dbc 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 12 bdev_ba54d78c-a118- mypool myimage12 8 GiB 512 Bytes ba54d78c-a118-4885- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 4885-be91-cc8c4c33830f be91-cc8c4c33830f 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 13 bdev_189c7a10-b33e- mypool myimage13 8 GiB 512 Bytes 189c7a10-b33e-4bb9- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 4bb9-9a52-4e1b81407374 9a52-4e1b81407374 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 14 bdev_06e2ef3a-ca82- mypool myimage14 8 GiB 512 Bytes 06e2ef3a-ca82-4bcb- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.302 INFO:tasks.workunit.client.0.smithi038.stdout: 4bcb-8675-e2f9559d0645 8675-e2f9559d0645 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 15 bdev_849c8403-b862- mypool myimage15 8 GiB 512 Bytes 849c8403-b862-46f7- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 46f7-82e8-a19b0a97dd90 82e8-a19b0a97dd90 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 16 bdev_1ec594ff-1157- mypool myimage16 8 GiB 512 Bytes 1ec594ff-1157-4225- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 4225-98ec-c6c41b32344b 98ec-c6c41b32344b 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 17 bdev_02593951-8f1d- mypool myimage17 8 GiB 512 Bytes 02593951-8f1d-487f- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 487f-8486-78cf08ee9025 8486-78cf08ee9025 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 18 bdev_e304d724-9f62- mypool myimage18 8 GiB 512 Bytes e304d724-9f62-459b- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 459b-b789-7137b80a0618 b789-7137b80a0618 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 19 bdev_e0592f88-f459- mypool myimage19 8 GiB 512 Bytes e0592f88-f459-49b5- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 49b5-b269-949b89c731f7 b269-949b89c731f7 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 20 bdev_42eb2216-ff26- mypool myimage20 8 GiB 512 Bytes 42eb2216-ff26-4390- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:47.303 INFO:tasks.workunit.client.0.smithi038.stdout: 4390-90a3-97b35df81f87 90a3-97b35df81f87 2024-09-02T12:05:47.594 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:47 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:47] INFO grpc.py:1212 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode1, context: , client address: IPv4 172.21.15.38:52870 2024-09-02T12:05:48.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:47 smithi087 ceph-mon[21100]: pgmap v365: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 1.3 KiB/s wr, 51 op/s 2024-09-02T12:05:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:47 smithi064 ceph-mon[17163]: pgmap v365: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 1.3 KiB/s wr, 51 op/s 2024-09-02T12:05:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:47 smithi094 ceph-mon[20404]: pgmap v365: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 140 KiB/s rd, 1.3 KiB/s wr, 51 op/s 2024-09-02T12:05:48.129 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:05:48.129 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:48.129 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 --format plain namespace list --subsystem nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:48.202 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:48Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:49.092 INFO:tasks.workunit.client.0.smithi038.stdout:Namespaces in subsystem nqn.2016-06.io.spdk:cnode2: 2024-09-02T12:05:49.092 INFO:tasks.workunit.client.0.smithi038.stdout: NSID Bdev RBD RBD Image Block UUID Load R/W IOs R/W MBs Read MBs Write MBs 2024-09-02T12:05:49.092 INFO:tasks.workunit.client.0.smithi038.stdout: Name Pool Image Size Size Balancing per per per per 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: Group second second second second 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: 1 bdev_e68b47f7-f3f5- mypool myimage21 8 GiB 512 Bytes e68b47f7-f3f5-4ba6- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: 4ba6-aff8-bae6a4b751b0 aff8-bae6a4b751b0 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: 2 bdev_840dc038-a66d- mypool myimage22 8 GiB 512 Bytes 840dc038-a66d-4c9e- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: 4c9e-9bf9-15d7e098ba19 9bf9-15d7e098ba19 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: 3 bdev_bcef3a86-f7ac- mypool myimage23 8 GiB 512 Bytes bcef3a86-f7ac-46c8- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.093 INFO:tasks.workunit.client.0.smithi038.stdout: 46c8-8c65-95ef522f586f 8c65-95ef522f586f 2024-09-02T12:05:49.094 INFO:tasks.workunit.client.0.smithi038.stdout: 4 bdev_52a33712-7b0e- mypool myimage24 8 GiB 512 Bytes 52a33712-7b0e-4b92- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.094 INFO:tasks.workunit.client.0.smithi038.stdout: 4b92-abad-4bd68e7cb304 abad-4bd68e7cb304 2024-09-02T12:05:49.094 INFO:tasks.workunit.client.0.smithi038.stdout: 5 bdev_b9ca847d-4dcb- mypool myimage25 8 GiB 512 Bytes b9ca847d-4dcb-441d- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.094 INFO:tasks.workunit.client.0.smithi038.stdout: 441d-a845-22a88dd680cc a845-22a88dd680cc 2024-09-02T12:05:49.094 INFO:tasks.workunit.client.0.smithi038.stdout: 6 bdev_5cc2b808-2dca- mypool myimage26 8 GiB 512 Bytes 5cc2b808-2dca-4ff9- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.094 INFO:tasks.workunit.client.0.smithi038.stdout: 4ff9-814b-967d94ee695f 814b-967d94ee695f 2024-09-02T12:05:49.095 INFO:tasks.workunit.client.0.smithi038.stdout: 7 bdev_d78cb720-fa0a- mypool myimage27 8 GiB 512 Bytes d78cb720-fa0a-4336- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.095 INFO:tasks.workunit.client.0.smithi038.stdout: 4336-9914-d8a99e821ef9 9914-d8a99e821ef9 2024-09-02T12:05:49.095 INFO:tasks.workunit.client.0.smithi038.stdout: 8 bdev_bdeff7fa-3026- mypool myimage28 8 GiB 512 Bytes bdeff7fa-3026-43ed- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.095 INFO:tasks.workunit.client.0.smithi038.stdout: 43ed-a9d1-845f00917b5d a9d1-845f00917b5d 2024-09-02T12:05:49.095 INFO:tasks.workunit.client.0.smithi038.stdout: 9 bdev_88e9aca1-c840- mypool myimage29 8 GiB 512 Bytes 88e9aca1-c840-4d93- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.095 INFO:tasks.workunit.client.0.smithi038.stdout: 4d93-9444-7a339bb07efb 9444-7a339bb07efb 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 10 bdev_bfcbf073-8eab- mypool myimage30 8 GiB 512 Bytes bfcbf073-8eab-4726- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 4726-8aa5-9616df85bee8 8aa5-9616df85bee8 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 11 bdev_2b28c17b-650d- mypool myimage31 8 GiB 512 Bytes 2b28c17b-650d-4aad- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 4aad-aa11-b432947561d5 aa11-b432947561d5 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 12 bdev_e25a419f-d57e- mypool myimage32 8 GiB 512 Bytes e25a419f-d57e-4dfb- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 4dfb-a465-d40da05e44c5 a465-d40da05e44c5 2024-09-02T12:05:49.096 INFO:tasks.workunit.client.0.smithi038.stdout: 13 bdev_e7199db1-0e14- mypool myimage33 8 GiB 512 Bytes e7199db1-0e14-47dc- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.097 INFO:tasks.workunit.client.0.smithi038.stdout: 47dc-93cb-6cd97484f3b4 93cb-6cd97484f3b4 2024-09-02T12:05:49.097 INFO:tasks.workunit.client.0.smithi038.stdout: 14 bdev_a995fbc3-9dbc- mypool myimage34 8 GiB 512 Bytes a995fbc3-9dbc-43b9- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.097 INFO:tasks.workunit.client.0.smithi038.stdout: 43b9-ab21-579813796890 ab21-579813796890 2024-09-02T12:05:49.097 INFO:tasks.workunit.client.0.smithi038.stdout: 15 bdev_b8506057-a178- mypool myimage35 8 GiB 512 Bytes b8506057-a178-4f80- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.097 INFO:tasks.workunit.client.0.smithi038.stdout: 4f80-a6be-0b5fc0a07487 a6be-0b5fc0a07487 2024-09-02T12:05:49.097 INFO:tasks.workunit.client.0.smithi038.stdout: 16 bdev_63cc09fa-ece0- mypool myimage36 8 GiB 512 Bytes 63cc09fa-ece0-493e- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 493e-8a42-19c8f5106cf2 8a42-19c8f5106cf2 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 17 bdev_73c62527-dd34- mypool myimage37 8 GiB 512 Bytes 73c62527-dd34-41cf- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 41cf-8df2-f51afb05cec0 8df2-f51afb05cec0 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 18 bdev_1d3b615a-3ba9- mypool myimage38 8 GiB 512 Bytes 1d3b615a-3ba9-48ba- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 48ba-845e-70403d71658f 845e-70403d71658f 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 19 bdev_755c5504-369c- mypool myimage39 8 GiB 512 Bytes 755c5504-369c-43fc- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.098 INFO:tasks.workunit.client.0.smithi038.stdout: 43fc-9da2-ccd5a6361150 9da2-ccd5a6361150 2024-09-02T12:05:49.099 INFO:tasks.workunit.client.0.smithi038.stdout: 20 bdev_0768e8b5-df41- mypool myimage40 8 GiB 512 Bytes 0768e8b5-df41-497d- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:49.099 INFO:tasks.workunit.client.0.smithi038.stdout: 497d-9343-ac44cee11a6a 9343-ac44cee11a6a 2024-09-02T12:05:49.344 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:48 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:48] INFO grpc.py:1212 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode2, context: , client address: IPv4 172.21.15.38:52724 2024-09-02T12:05:49.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:49.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:49.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:49.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:49.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:49.659 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:49.692 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:49.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:49.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:49.869 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:49.890 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:49 smithi087 ceph-mon[21100]: pgmap v366: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 853 B/s wr, 42 op/s 2024-09-02T12:05:49.900 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:49.922 INFO:tasks.workunit.client.0.smithi038.stderr:+ for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT) 2024-09-02T12:05:49.922 INFO:tasks.workunit.client.0.smithi038.stderr:+ subsystem_nqn=nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:49.922 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo podman run -it quay.io/ceph/nvmeof-cli:1.2 --server-address 172.21.15.64 --server-port 5500 --format plain namespace list --subsystem nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:49.931 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:49 smithi094 ceph-mon[20404]: pgmap v366: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 853 B/s wr, 42 op/s 2024-09-02T12:05:49.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:49.952 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:49 smithi064 ceph-mon[17163]: pgmap v366: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 853 B/s wr, 42 op/s 2024-09-02T12:05:49.995 INFO:tasks.workunit.client.0.smithi038.stderr:time="2024-09-02T12:05:49Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2024-09-02T12:05:50.908 INFO:tasks.workunit.client.0.smithi038.stdout:Namespaces in subsystem nqn.2016-06.io.spdk:cnode3: 2024-09-02T12:05:50.908 INFO:tasks.workunit.client.0.smithi038.stdout: NSID Bdev RBD RBD Image Block UUID Load R/W IOs R/W MBs Read MBs Write MBs 2024-09-02T12:05:50.908 INFO:tasks.workunit.client.0.smithi038.stdout: Name Pool Image Size Size Balancing per per per per 2024-09-02T12:05:50.908 INFO:tasks.workunit.client.0.smithi038.stdout: Group second second second second 2024-09-02T12:05:50.908 INFO:tasks.workunit.client.0.smithi038.stdout: 1 bdev_4bcf6c93-7460- mypool myimage41 8 GiB 512 Bytes 4bcf6c93-7460-4a42- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.908 INFO:tasks.workunit.client.0.smithi038.stdout: 4a42-8ec4-a048bf9f7108 8ec4-a048bf9f7108 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 2 bdev_69fc3401-6585- mypool myimage42 8 GiB 512 Bytes 69fc3401-6585-446a- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 446a-a9cf-f15cf86bf365 a9cf-f15cf86bf365 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 3 bdev_d96e1dcc-4b70- mypool myimage43 8 GiB 512 Bytes d96e1dcc-4b70-4b49- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 4b49-9e6a-63c891acca04 9e6a-63c891acca04 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 4 bdev_6d3e4936-9e03- mypool myimage44 8 GiB 512 Bytes 6d3e4936-9e03-425b- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 425b-ba93-b65323d44402 ba93-b65323d44402 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 5 bdev_547cce68-47ad- mypool myimage45 8 GiB 512 Bytes 547cce68-47ad-4227- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 4227-a432-c48eb15be142 a432-c48eb15be142 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 6 bdev_99e45781-7fdb- mypool myimage46 8 GiB 512 Bytes 99e45781-7fdb-4b7e- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 4b7e-8dfa-55a8cd1396b8 8dfa-55a8cd1396b8 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 7 bdev_58f2df23-338d- mypool myimage47 8 GiB 512 Bytes 58f2df23-338d-49b1- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 49b1-bba5-6eee6231dbf9 bba5-6eee6231dbf9 2024-09-02T12:05:50.909 INFO:tasks.workunit.client.0.smithi038.stdout: 8 bdev_6fb4bc29-355c- mypool myimage48 8 GiB 512 Bytes 6fb4bc29-355c-4b03- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 4b03-9a1a-9c91e407d92e 9a1a-9c91e407d92e 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 9 bdev_0aa81276-d941- mypool myimage49 8 GiB 512 Bytes 0aa81276-d941-465a- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 465a-90c0-bb35fb8bbf0f 90c0-bb35fb8bbf0f 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 10 bdev_4e4b44e6-d095- mypool myimage50 8 GiB 512 Bytes 4e4b44e6-d095-421d- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 421d-a9ba-35a28fd6faef a9ba-35a28fd6faef 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 11 bdev_40f6a294-315b- mypool myimage51 8 GiB 512 Bytes 40f6a294-315b-40eb- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 40eb-9839-236059ddbec0 9839-236059ddbec0 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 12 bdev_3bb2e008-3e5e- mypool myimage52 8 GiB 512 Bytes 3bb2e008-3e5e-4ee7- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 4ee7-bbe3-498688b9775f bbe3-498688b9775f 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 13 bdev_b5597828-c750- mypool myimage53 8 GiB 512 Bytes b5597828-c750-4702- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 4702-aa14-9709fb7d1d91 aa14-9709fb7d1d91 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 14 bdev_70de1546-c18d- mypool myimage54 8 GiB 512 Bytes 70de1546-c18d-49bb- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 49bb-b824-32b6b807a5ab b824-32b6b807a5ab 2024-09-02T12:05:50.910 INFO:tasks.workunit.client.0.smithi038.stdout: 15 bdev_d019f375-0d32- mypool myimage55 8 GiB 512 Bytes d019f375-0d32-4e88- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 4e88-9dfc-eecb097fcb29 9dfc-eecb097fcb29 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 16 bdev_bc1c35d3-fccd- mypool myimage56 8 GiB 512 Bytes bc1c35d3-fccd-4a25- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 4a25-91b7-b79512fd3ce3 91b7-b79512fd3ce3 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 17 bdev_3f319486-9f14- mypool myimage57 8 GiB 512 Bytes 3f319486-9f14-4aa1- 2 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 4aa1-a409-a33fcca2ec41 a409-a33fcca2ec41 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 18 bdev_da5a06d9-7937- mypool myimage58 8 GiB 512 Bytes da5a06d9-7937-4c24- 3 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 4c24-9b2b-40e24f9c74d8 9b2b-40e24f9c74d8 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 19 bdev_3a7171f0-c316- mypool myimage59 8 GiB 512 Bytes 3a7171f0-c316-4798- 4 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 4798-8db4-433cac4e7b05 8db4-433cac4e7b05 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 20 bdev_22f8f0f6-9933- mypool myimage60 8 GiB 512 Bytes 22f8f0f6-9933-4a95- 1 unlimited unlimited unlimited unlimited 2024-09-02T12:05:50.911 INFO:tasks.workunit.client.0.smithi038.stdout: 4a95-9582-0f374f94cad7 9582-0f374f94cad7 2024-09-02T12:05:51.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:05:50 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:05:50] INFO grpc.py:1212 (2): Received request to list all namespaces for nqn.2016-06.io.spdk:cnode3, context: , client address: IPv4 172.21.15.38:49794 2024-09-02T12:05:51.740 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Subsystem setup done 2024-09-02T12:05:51.741 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Subsystem setup done' 2024-09-02T12:05:51.741 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-02T12:05:51.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-02T12:05:51.828 INFO:tasks.workunit:Running workunits matching nvmeof/basic_tests.sh on client.0... 2024-09-02T12:05:51.830 INFO:tasks.workunit:Running workunit nvmeof/basic_tests.sh... 2024-09-02T12:05:51.830 DEBUG:teuthology.orchestra.run.smithi038:workunit test nvmeof/basic_tests.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_IMAGE_PREFIX=myimage RBD_POOL=mypool timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/basic_tests.sh 2024-09-02T12:05:51.848 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo modprobe nvme-fabrics 2024-09-02T12:05:51.870 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo modprobe nvme-tcp 2024-09-02T12:05:51.900 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo dnf reinstall nvme-cli -y 2024-09-02T12:05:52.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:51 smithi087 ceph-mon[21100]: pgmap v367: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 682 B/s wr, 26 op/s 2024-09-02T12:05:52.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:51 smithi064 ceph-mon[17163]: pgmap v367: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 682 B/s wr, 26 op/s 2024-09-02T12:05:52.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:51 smithi094 ceph-mon[20404]: pgmap v367: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 682 B/s wr, 26 op/s 2024-09-02T12:05:52.452 INFO:tasks.workunit.client.0.smithi038.stdout:Last metadata expiration check: 0:15:04 ago on Mon 02 Sep 2024 11:50:48 AM UTC. 2024-09-02T12:05:52.589 INFO:tasks.workunit.client.0.smithi038.stdout:Dependencies resolved. 2024-09-02T12:05:52.590 INFO:tasks.workunit.client.0.smithi038.stdout:================================================================================ 2024-09-02T12:05:52.590 INFO:tasks.workunit.client.0.smithi038.stdout: Package Architecture Version Repository Size 2024-09-02T12:05:52.590 INFO:tasks.workunit.client.0.smithi038.stdout:================================================================================ 2024-09-02T12:05:52.590 INFO:tasks.workunit.client.0.smithi038.stdout:Reinstalling: 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout:Transaction Summary 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout:================================================================================ 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout:Total download size: 906 k 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout:Installed size: 5.3 M 2024-09-02T12:05:52.591 INFO:tasks.workunit.client.0.smithi038.stdout:Downloading Packages: 2024-09-02T12:05:52.886 INFO:tasks.workunit.client.0.smithi038.stdout:nvme-cli-2.9.1-6.el9.x86_64.rpm 4.8 MB/s | 906 kB 00:00 2024-09-02T12:05:52.886 INFO:tasks.workunit.client.0.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-02T12:05:52.886 INFO:tasks.workunit.client.0.smithi038.stdout:Total 3.0 MB/s | 906 kB 00:00 2024-09-02T12:05:52.903 INFO:tasks.workunit.client.0.smithi038.stdout:Running transaction check 2024-09-02T12:05:52.914 INFO:tasks.workunit.client.0.smithi038.stdout:Transaction check succeeded. 2024-09-02T12:05:52.914 INFO:tasks.workunit.client.0.smithi038.stdout:Running transaction test 2024-09-02T12:05:53.010 INFO:tasks.workunit.client.0.smithi038.stdout:Transaction test succeeded. 2024-09-02T12:05:53.011 INFO:tasks.workunit.client.0.smithi038.stdout:Running transaction 2024-09-02T12:05:53.454 INFO:tasks.workunit.client.0.smithi038.stdout: Preparing : 1/1 2024-09-02T12:05:53.547 INFO:tasks.workunit.client.0.smithi038.stdout: Reinstalling : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-02T12:05:53.848 INFO:tasks.workunit.client.0.smithi038.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-02T12:05:53.955 INFO:tasks.workunit.client.0.smithi038.stdout: Cleanup : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-02T12:05:54.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:53 smithi087 ceph-mon[21100]: pgmap v368: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 170 B/s wr, 19 op/s 2024-09-02T12:05:54.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:53 smithi064 ceph-mon[17163]: pgmap v368: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 170 B/s wr, 19 op/s 2024-09-02T12:05:54.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:53 smithi094 ceph-mon[20404]: pgmap v368: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 170 B/s wr, 19 op/s 2024-09-02T12:05:54.720 INFO:tasks.workunit.client.0.smithi038.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-02T12:05:54.721 INFO:tasks.workunit.client.0.smithi038.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-02T12:05:54.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:05:54.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:05:55.056 INFO:tasks.workunit.client.0.smithi038.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-02T12:05:55.056 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-09-02T12:05:55.056 INFO:tasks.workunit.client.0.smithi038.stdout:Reinstalled: 2024-09-02T12:05:55.056 INFO:tasks.workunit.client.0.smithi038.stdout: nvme-cli-2.9.1-6.el9.x86_64 2024-09-02T12:05:55.056 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-09-02T12:05:55.056 INFO:tasks.workunit.client.0.smithi038.stdout:Complete! 2024-09-02T12:05:55.063 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:05:55.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:05:55.166 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:05:55.237 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:05:55.269 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:05:55.343 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:05:55.374 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo lsmod 2024-09-02T12:05:55.374 INFO:tasks.workunit.client.0.smithi038.stderr:+ grep nvme 2024-09-02T12:05:55.389 INFO:tasks.workunit.client.0.smithi038.stdout:nvme_tcp 57344 0 2024-09-02T12:05:55.389 INFO:tasks.workunit.client.0.smithi038.stdout:nvme_keyring 16384 1 nvme_tcp 2024-09-02T12:05:55.389 INFO:tasks.workunit.client.0.smithi038.stdout:nvme_fabrics 45056 1 nvme_tcp 2024-09-02T12:05:55.389 INFO:tasks.workunit.client.0.smithi038.stdout:nvme 65536 5 2024-09-02T12:05:55.389 INFO:tasks.workunit.client.0.smithi038.stdout:nvme_core 237568 8 nvme_tcp,nvme,nvme_fabrics 2024-09-02T12:05:55.389 INFO:tasks.workunit.client.0.smithi038.stdout:nvme_auth 28672 1 nvme_core 2024-09-02T12:05:55.390 INFO:tasks.workunit.client.0.smithi038.stdout:t10_pi 20480 2 sd_mod,nvme_core 2024-09-02T12:05:55.390 INFO:tasks.workunit.client.0.smithi038.stderr:+ nvme version 2024-09-02T12:05:55.390 INFO:tasks.workunit.client.0.smithi038.stdout:nvme version 2.9.1 (git 2.9.1) 2024-09-02T12:05:55.390 INFO:tasks.workunit.client.0.smithi038.stdout:libnvme version 1.9 (git 1.9) 2024-09-02T12:05:55.391 INFO:tasks.workunit.client.0.smithi038.stderr:+ source /etc/ceph/nvmeof.env 2024-09-02T12:05:55.391 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_GATEWAY_IP_ADDRESSES=172.21.15.64,172.21.15.87,172.21.15.94,172.21.15.99 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_GATEWAY_NAMES=smithi064,smithi087,smithi094,smithi099 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS=172.21.15.64 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_CLI_IMAGE=quay.io/ceph/nvmeof-cli:1.2 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_SUBSYSTEMS_PREFIX=nqn.2016-06.io.spdk:cnode 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_SUBSYSTEMS_COUNT=3 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_NAMESPACES_COUNT=20 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_PORT=4420 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:++ NVMEOF_SRPORT=5500 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ SPDK_CONTROLLER='Ceph bdev Controller' 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ DISCOVERY_PORT=8009 2024-09-02T12:05:55.392 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run disconnect_all 2024-09-02T12:05:55.393 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: disconnect_all 2024-09-02T12:05:55.393 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: disconnect_all' 2024-09-02T12:05:55.393 INFO:tasks.workunit.client.0.smithi038.stderr:+ disconnect_all 2024-09-02T12:05:55.393 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo nvme disconnect-all 2024-09-02T12:05:55.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:05:55.416 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list 2024-09-02T12:05:55.426 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='Node Generic SN Model Namespace Usage Format FW Rev 2024-09-02T12:05:55.426 INFO:tasks.workunit.client.0.smithi038.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2024-09-02T12:05:55.426 INFO:tasks.workunit.client.0.smithi038.stderr:/dev/nvme0n1 /dev/ng0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 0x1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D' 2024-09-02T12:05:55.426 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Node Generic SN Model Namespace Usage Format FW Rev 2024-09-02T12:05:55.427 INFO:tasks.workunit.client.0.smithi038.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2024-09-02T12:05:55.427 INFO:tasks.workunit.client.0.smithi038.stderr:/dev/nvme0n1 /dev/ng0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 0x1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D' 2024-09-02T12:05:55.427 INFO:tasks.workunit.client.0.smithi038.stderr:+ grep -q 'Ceph bdev Controller' 2024-09-02T12:05:55.427 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:05:55.427 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] disconnect_all test passed! 2024-09-02T12:05:55.428 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] disconnect_all test passed!' 2024-09-02T12:05:55.428 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run discovery 2024-09-02T12:05:55.428 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: discovery 2024-09-02T12:05:55.429 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: discovery' 2024-09-02T12:05:55.429 INFO:tasks.workunit.client.0.smithi038.stderr:+ discovery 2024-09-02T12:05:55.429 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme discover -t tcp -a 172.21.15.64 -s 8009 2024-09-02T12:05:55.446 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:+ output=' 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:Discovery Log Number of Records 12, Generation counter 1 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 0====== 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 0 2024-09-02T12:05:55.452 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.64 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 1====== 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 1 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.453 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.87 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 2====== 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 2 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.94 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.454 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 3====== 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 3 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.99 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 4====== 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.455 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 4 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.64 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 5====== 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.456 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 5 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.87 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 6====== 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 6 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.457 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.94 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 7====== 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 7 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.99 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.458 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 8====== 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 8 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.64 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 9====== 2024-09-02T12:05:55.459 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 9 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.87 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 10====== 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.460 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 10 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.94 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 11====== 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.461 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 11 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.99 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none' 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:+ expected_discovery_stdout='subtype: nvme subsystem' 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo ' 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:Discovery Log Number of Records 12, Generation counter 1 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 0====== 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.462 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 0 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.64 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 1====== 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 1 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.463 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.87 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 2====== 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 2 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.94 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.464 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 3====== 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 3 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.99 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 4====== 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.465 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 4 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.64 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 5====== 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.466 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 5 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.87 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 6====== 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 6 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.467 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.94 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 7====== 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 7 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode2 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.99 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.468 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 8====== 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 8 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.64 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 9====== 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.469 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 9 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.87 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 10====== 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.470 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 10 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.94 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:=====Discovery Log Entry 11====== 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:trtype: tcp 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:adrfam: ipv4 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:subtype: nvme subsystem 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:treq: not required 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:portid: 11 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:trsvcid: 4420 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:subnqn: nqn.2016-06.io.spdk:cnode3 2024-09-02T12:05:55.471 INFO:tasks.workunit.client.0.smithi038.stderr:traddr: 172.21.15.99 2024-09-02T12:05:55.472 INFO:tasks.workunit.client.0.smithi038.stderr:eflags: none 2024-09-02T12:05:55.472 INFO:tasks.workunit.client.0.smithi038.stderr:sectype: none' 2024-09-02T12:05:55.472 INFO:tasks.workunit.client.0.smithi038.stderr:+ grep -q 'subtype: nvme subsystem' 2024-09-02T12:05:55.472 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:05:55.472 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] discovery test passed! 2024-09-02T12:05:55.473 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] discovery test passed!' 2024-09-02T12:05:55.473 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run connect 2024-09-02T12:05:55.473 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: connect 2024-09-02T12:05:55.474 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: connect' 2024-09-02T12:05:55.474 INFO:tasks.workunit.client.0.smithi038.stderr:+ connect 2024-09-02T12:05:55.474 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo nvme connect -t tcp --traddr 172.21.15.64 -s 4420 -n nqn.2016-06.io.spdk:cnode1 2024-09-02T12:05:55.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:05:55.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:05:55.546 INFO:tasks.workunit.client.0.smithi038.stdout:connecting to device: nvme1 2024-09-02T12:05:55.547 INFO:tasks.workunit.client.0.smithi038.stderr:+ sleep 5 2024-09-02T12:05:55.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:55 smithi087 ceph-mon[21100]: pgmap v369: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-02T12:05:55.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:55 smithi094 ceph-mon[20404]: pgmap v369: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-02T12:05:56.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:55 smithi064 ceph-mon[17163]: pgmap v369: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 85 B/s wr, 11 op/s 2024-09-02T12:05:58.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:57 smithi087 ceph-mon[21100]: pgmap v370: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-02T12:05:58.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:57 smithi064 ceph-mon[17163]: pgmap v370: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-02T12:05:58.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:57 smithi094 ceph-mon[20404]: pgmap v370: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-02T12:06:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:05:59 smithi087 ceph-mon[21100]: pgmap v371: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 426 B/s wr, 3 op/s 2024-09-02T12:06:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:05:59 smithi064 ceph-mon[17163]: pgmap v371: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 426 B/s wr, 3 op/s 2024-09-02T12:06:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:05:59 smithi094 ceph-mon[20404]: pgmap v371: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 426 B/s wr, 3 op/s 2024-09-02T12:06:00.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:00.549 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list --output-format=json 2024-09-02T12:06:00.567 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='{ 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "Devices":[ 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n8", 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n8", 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.568 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n4", 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n4", 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.569 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n20", 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n20", 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.570 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n16", 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n16", 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.571 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n12", 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n12", 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.572 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme0n1", 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng0n1", 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"QDV1013D", 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"INTEL SSDPEDKE040T7", 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"BTLE736103BB4P0KGN", 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":4000787030016, 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":7814037168, 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":4000787030016, 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.573 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr:}' 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '{ 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "Devices":[ 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n8", 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n8", 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.574 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n4", 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n4", 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.575 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n20", 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n20", 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.576 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n16", 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n16", 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.577 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme1n12", 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng1n12", 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.578 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme0n1", 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng0n1", 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"QDV1013D", 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"INTEL SSDPEDKE040T7", 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"BTLE736103BB4P0KGN", 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":4000787030016, 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":7814037168, 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":4000787030016, 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:00.579 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.580 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.580 INFO:tasks.workunit.client.0.smithi038.stderr:}' 2024-09-02T12:06:00.580 INFO:tasks.workunit.client.0.smithi038.stderr:+ grep -q 'Ceph bdev Controller' 2024-09-02T12:06:00.580 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:06:00.581 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] connect test passed! 2024-09-02T12:06:00.581 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] connect test passed!' 2024-09-02T12:06:00.582 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run list_subsys 1 2024-09-02T12:06:00.582 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: list_subsys 2024-09-02T12:06:00.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:00.583 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: list_subsys' 2024-09-02T12:06:00.583 INFO:tasks.workunit.client.0.smithi038.stderr:+ list_subsys 1 2024-09-02T12:06:00.583 INFO:tasks.workunit.client.0.smithi038.stderr:+ expected_count=1 2024-09-02T12:06:00.583 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list-subsys --output-format=json 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='[ 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4", 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c", 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "Subsystems":[ 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys1", 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:00.584 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme1", 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys0", 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2014.08.org.nvmexpress:80868086BTLE736103BB4P0KGN INTEL SSDPEDKE040T7", 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.585 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme0", 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"pcie", 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"0000:02:00.0", 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.586 INFO:tasks.workunit.client.0.smithi038.stderr:]' 2024-09-02T12:06:00.587 INFO:tasks.workunit.client.0.smithi038.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4",' '"HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c",' '"Subsystems":[' '{' '"Name":"nvme-subsys1",' '"NQN":"nqn.2016-06.io.spdk:cnode1",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme1",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2014.08.org.nvmexpress:80868086BTLE736103BB4P0KGN' INTEL 'SSDPEDKE040T7",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2024-09-02T12:06:00.587 INFO:tasks.workunit.client.0.smithi038.stderr:++ grep -o '"tcp"' 2024-09-02T12:06:00.588 INFO:tasks.workunit.client.0.smithi038.stderr:++ wc -l 2024-09-02T12:06:00.588 INFO:tasks.workunit.client.0.smithi038.stderr:+ multipath=1 2024-09-02T12:06:00.588 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 1 -ne 1 ']' 2024-09-02T12:06:00.588 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:06:00.588 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] list_subsys test passed! 2024-09-02T12:06:00.588 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] list_subsys test passed!' 2024-09-02T12:06:00.589 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run disconnect_all 2024-09-02T12:06:00.589 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: disconnect_all 2024-09-02T12:06:00.589 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: disconnect_all' 2024-09-02T12:06:00.589 INFO:tasks.workunit.client.0.smithi038.stderr:+ disconnect_all 2024-09-02T12:06:00.589 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo nvme disconnect-all 2024-09-02T12:06:00.652 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:00.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:00.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:00.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:00.859 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:00.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:00.934 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='Node Generic SN Model Namespace Usage Format FW Rev 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:/dev/nvme0n1 /dev/ng0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 0x1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D' 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo 'Node Generic SN Model Namespace Usage Format FW Rev 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:/dev/nvme0n1 /dev/ng0n1 BTLE736103BB4P0KGN INTEL SSDPEDKE040T7 0x1 4.00 TB / 4.00 TB 512 B + 0 B QDV1013D' 2024-09-02T12:06:00.948 INFO:tasks.workunit.client.0.smithi038.stderr:+ grep -q 'Ceph bdev Controller' 2024-09-02T12:06:00.949 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:06:00.949 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] disconnect_all test passed! 2024-09-02T12:06:00.950 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] disconnect_all test passed!' 2024-09-02T12:06:00.950 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run list_subsys 0 2024-09-02T12:06:00.950 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: list_subsys 2024-09-02T12:06:00.951 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: list_subsys' 2024-09-02T12:06:00.951 INFO:tasks.workunit.client.0.smithi038.stderr:+ list_subsys 0 2024-09-02T12:06:00.951 INFO:tasks.workunit.client.0.smithi038.stderr:+ expected_count=0 2024-09-02T12:06:00.951 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list-subsys --output-format=json 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='[ 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4", 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c", 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "Subsystems":[ 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys0", 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2014.08.org.nvmexpress:80868086BTLE736103BB4P0KGN INTEL SSDPEDKE040T7", 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:00.963 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme0", 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"pcie", 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"0000:02:00.0", 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:00.964 INFO:tasks.workunit.client.0.smithi038.stderr:]' 2024-09-02T12:06:00.965 INFO:tasks.workunit.client.0.smithi038.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4",' '"HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c",' '"Subsystems":[' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2014.08.org.nvmexpress:80868086BTLE736103BB4P0KGN' INTEL 'SSDPEDKE040T7",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2024-09-02T12:06:00.965 INFO:tasks.workunit.client.0.smithi038.stderr:++ grep -o '"tcp"' 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stderr:++ wc -l 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stderr:+ multipath=0 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -ne 0 ']' 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] list_subsys test passed!' 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] list_subsys test passed! 2024-09-02T12:06:00.966 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run connect_all 2024-09-02T12:06:00.967 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: connect_all 2024-09-02T12:06:00.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: connect_all' 2024-09-02T12:06:00.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ connect_all 2024-09-02T12:06:00.967 INFO:tasks.workunit.client.0.smithi038.stderr:+ sudo nvme connect-all --traddr=172.21.15.64 --transport=tcp -l 3600 2024-09-02T12:06:01.005 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:01.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:01.067 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:01.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:02.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:01 smithi087 ceph-mon[21100]: pgmap v372: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 426 B/s wr, 5 op/s 2024-09-02T12:06:02.055 INFO:tasks.workunit.client.0.smithi038.stderr:+ sleep 5 2024-09-02T12:06:02.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:01 smithi064 ceph-mon[17163]: pgmap v372: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 426 B/s wr, 5 op/s 2024-09-02T12:06:02.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:01 smithi094 ceph-mon[20404]: pgmap v372: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 426 B/s wr, 5 op/s 2024-09-02T12:06:04.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:03 smithi087 ceph-mon[21100]: pgmap v373: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-02T12:06:04.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:03 smithi064 ceph-mon[17163]: pgmap v373: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-02T12:06:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:03 smithi094 ceph-mon[20404]: pgmap v373: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-02T12:06:06.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:05 smithi087 ceph-mon[21100]: pgmap v374: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-09-02T12:06:06.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:05 smithi064 ceph-mon[17163]: pgmap v374: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-09-02T12:06:06.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:05 smithi094 ceph-mon[20404]: pgmap v374: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-09-02T12:06:06.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:06.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:06.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:06.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:06.339 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:06.410 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:06.443 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:06.515 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:06.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:06.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:06.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:06.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:07.056 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list --output-format=json 2024-09-02T12:06:07.093 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='{ 2024-09-02T12:06:07.093 INFO:tasks.workunit.client.0.smithi038.stderr: "Devices":[ 2024-09-02T12:06:07.093 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.093 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":9, 2024-09-02T12:06:07.093 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n9", 2024-09-02T12:06:07.093 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n9", 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:07.094 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n8", 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n8", 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.095 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":7, 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n7", 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n7", 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.096 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":6, 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n6", 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n6", 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.097 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":5, 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n5", 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n5", 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.098 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n4", 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n4", 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.099 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":3, 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n3", 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n3", 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.100 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n20", 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n20", 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.101 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":2, 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n2", 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n2", 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.102 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":19, 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n19", 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n19", 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.103 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":18, 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n18", 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n18", 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.104 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":17, 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n17", 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n17", 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.105 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n16", 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n16", 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.106 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":15, 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n15", 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n15", 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.107 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":14, 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n14", 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n14", 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.108 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":13, 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n13", 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n13", 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.109 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n12", 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n12", 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.110 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":11, 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n11", 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n11", 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.111 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":10, 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n10", 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n10", 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.112 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n1", 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n1", 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.113 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":9, 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n9", 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n9", 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.114 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n8", 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n8", 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.115 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":7, 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n7", 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n7", 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.116 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":6, 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n6", 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n6", 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":5, 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n5", 2024-09-02T12:06:07.117 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n5", 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n4", 2024-09-02T12:06:07.118 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n4", 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":3, 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n3", 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n3", 2024-09-02T12:06:07.119 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n20", 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n20", 2024-09-02T12:06:07.120 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":2, 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n2", 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n2", 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.121 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":19, 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n19", 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n19", 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.122 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":18, 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n18", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n18", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.123 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":17, 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n17", 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n17", 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.124 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n16", 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n16", 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.125 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":15, 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n15", 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n15", 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.126 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":14, 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n14", 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n14", 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.127 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":13, 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n13", 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n13", 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.128 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n12", 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n12", 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.129 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":11, 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n11", 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n11", 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.130 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":10, 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n10", 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n10", 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n1", 2024-09-02T12:06:07.131 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n1", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":9, 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n9", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n9", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.132 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n8", 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n8", 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.133 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":7, 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n7", 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n7", 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.134 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":6, 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n6", 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n6", 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.135 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":5, 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n5", 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n5", 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.136 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n4", 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n4", 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.137 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":3, 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n3", 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n3", 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.138 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n20", 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n20", 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.139 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":2, 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n2", 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n2", 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.140 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":19, 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n19", 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n19", 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.141 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":18, 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n18", 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n18", 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.142 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":17, 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n17", 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n17", 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.143 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n16", 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n16", 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":15, 2024-09-02T12:06:07.144 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n15", 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n15", 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":14, 2024-09-02T12:06:07.145 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n14", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n14", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":13, 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n13", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n13", 2024-09-02T12:06:07.146 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n12", 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n12", 2024-09-02T12:06:07.147 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":11, 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n11", 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n11", 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.148 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":10, 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n10", 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n10", 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.149 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n1", 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n1", 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.150 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme0n1", 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng0n1", 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"QDV1013D", 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"INTEL SSDPEDKE040T7", 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"BTLE736103BB4P0KGN", 2024-09-02T12:06:07.151 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":4000787030016, 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":7814037168, 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":4000787030016, 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr:}' 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr:+ grep -q 'Ceph bdev Controller' 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '{ 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "Devices":[ 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":9, 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n9", 2024-09-02T12:06:07.152 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n9", 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n8", 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n8", 2024-09-02T12:06:07.153 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":7, 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n7", 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n7", 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.154 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":6, 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n6", 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n6", 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.155 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":5, 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n5", 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n5", 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.156 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n4", 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n4", 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.157 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":3, 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n3", 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n3", 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.158 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n20", 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n20", 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.159 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":2, 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n2", 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n2", 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.160 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":19, 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n19", 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n19", 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":18, 2024-09-02T12:06:07.161 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n18", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n18", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":17, 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n17", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n17", 2024-09-02T12:06:07.162 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n16", 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n16", 2024-09-02T12:06:07.163 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":15, 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n15", 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n15", 2024-09-02T12:06:07.164 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":14, 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n14", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n14", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.165 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":13, 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n13", 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n13", 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.166 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n12", 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n12", 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.167 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":11, 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n11", 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n11", 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.168 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":10, 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n10", 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n10", 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.169 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme6n1", 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng6n1", 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph61660518994858", 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.170 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":9, 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n9", 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n9", 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.171 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n8", 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n8", 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":7, 2024-09-02T12:06:07.172 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n7", 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n7", 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":6, 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n6", 2024-09-02T12:06:07.173 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n6", 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":5, 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n5", 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n5", 2024-09-02T12:06:07.174 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n4", 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n4", 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.175 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":3, 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n3", 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n3", 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.176 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n20", 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n20", 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.177 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":2, 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n2", 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n2", 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.178 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":19, 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n19", 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n19", 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.179 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":18, 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n18", 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n18", 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.180 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":17, 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n17", 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n17", 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.181 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n16", 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n16", 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":15, 2024-09-02T12:06:07.182 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n15", 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n15", 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":14, 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n14", 2024-09-02T12:06:07.183 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n14", 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":13, 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n13", 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n13", 2024-09-02T12:06:07.184 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n12", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n12", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.185 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":11, 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n11", 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n11", 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.186 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":10, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n10", 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n10", 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.187 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme2n1", 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng2n1", 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph81601849504533", 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.188 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":9, 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n9", 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n9", 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.189 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":8, 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n8", 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n8", 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.190 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":7, 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n7", 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n7", 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.191 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":6, 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n6", 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n6", 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":5, 2024-09-02T12:06:07.192 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n5", 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n5", 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":4, 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n4", 2024-09-02T12:06:07.193 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n4", 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":3, 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n3", 2024-09-02T12:06:07.194 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n3", 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":20, 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n20", 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n20", 2024-09-02T12:06:07.195 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":2, 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n2", 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n2", 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.196 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":19, 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n19", 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n19", 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.197 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":18, 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n18", 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n18", 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.198 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":17, 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n17", 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n17", 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.199 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":16, 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n16", 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n16", 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.200 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":15, 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n15", 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n15", 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.201 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":14, 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n14", 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n14", 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.202 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":13, 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n13", 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n13", 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.203 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":12, 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n12", 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n12", 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.204 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":11, 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n11", 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n11", 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.205 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":10, 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n10", 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n10", 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.206 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme10n1", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng10n1", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"24.01", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"Ceph bdev Controller", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"Ceph22728624249465", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":8589934592, 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":16777216, 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":8589934592, 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "NameSpace":1, 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "DevicePath":"/dev/nvme0n1", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "GenericPath":"/dev/ng0n1", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "Firmware":"QDV1013D", 2024-09-02T12:06:07.207 INFO:tasks.workunit.client.0.smithi038.stderr: "ModelNumber":"INTEL SSDPEDKE040T7", 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: "SerialNumber":"BTLE736103BB4P0KGN", 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: "UsedBytes":4000787030016, 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: "MaximumLBA":7814037168, 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: "PhysicalSize":4000787030016, 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: "SectorSize":512 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr:}' 2024-09-02T12:06:07.208 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:06:07.209 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] connect_all test passed! 2024-09-02T12:06:07.209 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] connect_all test passed!' 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stderr:++ echo 172.21.15.64,172.21.15.87,172.21.15.94,172.21.15.99 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stderr:++ tr -cd , 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stderr:++ wc -c 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stderr:+ gateways_count=4 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stderr:+ multipath_count=12 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stderr:+ test_run list_subsys 12 2024-09-02T12:06:07.210 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] Running test: list_subsys 2024-09-02T12:06:07.212 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] Running test: list_subsys' 2024-09-02T12:06:07.212 INFO:tasks.workunit.client.0.smithi038.stderr:+ list_subsys 12 2024-09-02T12:06:07.212 INFO:tasks.workunit.client.0.smithi038.stderr:+ expected_count=12 2024-09-02T12:06:07.212 INFO:tasks.workunit.client.0.smithi038.stderr:++ sudo nvme list-subsys --output-format=json 2024-09-02T12:06:07.212 INFO:tasks.workunit.client.0.smithi038.stderr:+ output='[ 2024-09-02T12:06:07.212 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "Subsystems":[ 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys6", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2016-06.io.spdk:cnode2", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme9", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.213 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme8", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme7", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.214 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme6", 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys2", 2024-09-02T12:06:07.215 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2016-06.io.spdk:cnode1", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme5", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme4", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.216 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme3", 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme2", 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.217 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys10", 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2016-06.io.spdk:cnode3", 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme13", 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.218 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme12", 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme11", 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.219 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme10", 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"tcp", 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38", 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: }, 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme-subsys0", 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "NQN":"nqn.2014.08.org.nvmexpress:80868086BTLE736103BB4P0KGN INTEL SSDPEDKE040T7", 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "IOPolicy":"numa", 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "Paths":[ 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: { 2024-09-02T12:06:07.220 INFO:tasks.workunit.client.0.smithi038.stderr: "Name":"nvme0", 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: "Transport":"pcie", 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: "Address":"0000:02:00.0", 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: "State":"live" 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: ] 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr: } 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr:]' 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr:++ echo '[' '{' '"HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4",' '"HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c",' '"Subsystems":[' '{' '"Name":"nvme-subsys6",' '"NQN":"nqn.2016-06.io.spdk:cnode2",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme9",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme8",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme7",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme6",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys2",' '"NQN":"nqn.2016-06.io.spdk:cnode1",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme5",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme4",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme3",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme2",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys10",' '"NQN":"nqn.2016-06.io.spdk:cnode3",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme13",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme12",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme11",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '},' '{' '"Name":"nvme10",' '"Transport":"tcp",' '"Address":"traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38",' '"State":"live"' '}' ']' '},' '{' '"Name":"nvme-subsys0",' '"NQN":"nqn.2014.08.org.nvmexpress:80868086BTLE736103BB4P0KGN' INTEL 'SSDPEDKE040T7",' '"IOPolicy":"numa",' '"Paths":[' '{' '"Name":"nvme0",' '"Transport":"pcie",' '"Address":"0000:02:00.0",' '"State":"live"' '}' ']' '}' ']' '}' ']' 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr:++ grep -o '"tcp"' 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr:++ wc -l 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr:+ multipath=12 2024-09-02T12:06:07.221 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 12 -ne 12 ']' 2024-09-02T12:06:07.222 INFO:tasks.workunit.client.0.smithi038.stderr:+ '[' 0 -eq 0 ']' 2024-09-02T12:06:07.222 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] list_subsys test passed! 2024-09-02T12:06:07.222 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] list_subsys test passed!' 2024-09-02T12:06:07.222 INFO:tasks.workunit.client.0.smithi038.stdout:-------------Test Summary------------- 2024-09-02T12:06:07.223 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '-------------Test Summary-------------' 2024-09-02T12:06:07.223 INFO:tasks.workunit.client.0.smithi038.stdout:[nvmeof] All nvmeof basic tests passed! 2024-09-02T12:06:07.224 INFO:tasks.workunit.client.0.smithi038.stderr:+ echo '[nvmeof] All nvmeof basic tests passed!' 2024-09-02T12:06:07.224 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-02T12:06:07.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-02T12:06:07.293 INFO:tasks.workunit:Stopping ['nvmeof/setup_subsystem.sh', 'nvmeof/basic_tests.sh'] on client.0... 2024-09-02T12:06:07.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-02T12:06:07.643 DEBUG:teuthology.parallel:result is None 2024-09-02T12:06:07.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-02T12:06:07.668 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-02T12:06:07.668 DEBUG:teuthology.orchestra.run.smithi038:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-02T12:06:07.735 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-02T12:06:07.735 INFO:teuthology.run_tasks:Running task nvmeof.thrash... 2024-09-02T12:06:07.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo nvme list --output-format=json | jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == "Ceph bdev Controller") | .DevicePath' 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n1 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n1 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n1 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n2 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n2 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n2 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n3 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n3 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n3 2024-09-02T12:06:07.813 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n4 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n4 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n4 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n5 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n5 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n5 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n6 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n6 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n6 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n7 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n7 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n7 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n8 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n8 2024-09-02T12:06:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n8 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n9 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n9 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n9 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n10 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n10 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n10 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n11 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n11 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n11 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n12 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n12 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n12 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n13 2024-09-02T12:06:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n13 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n13 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n14 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n14 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n14 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n15 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n15 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n15 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n16 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n16 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n16 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n17 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n17 2024-09-02T12:06:07.816 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n17 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n18 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n18 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n18 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n19 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n19 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n19 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme6n20 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme2n20 2024-09-02T12:06:07.817 INFO:teuthology.orchestra.run.smithi038.stdout:/dev/nvme10n20 2024-09-02T12:06:07.818 INFO:teuthology.run_tasks:Running task workunit... 2024-09-02T12:06:07.827 INFO:tasks.workunit:Pulling workunits from ref 6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 2024-09-02T12:06:07.827 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-02T12:06:07.828 DEBUG:teuthology.orchestra.run.smithi094:> stat -- /home/ubuntu/cephtest/mnt.3 2024-09-02T12:06:07.828 INFO:tasks.nvmeof.[nvmeof.thrasher]:start thrashing 2024-09-02T12:06:07.828 INFO:tasks.nvmeof.[nvmeof.thrasher]:seed: 1725278767, , max thrash delay: 90, min thrash delay: 60 max revive delay: 130, min revive delay: 100 daemons: 4 2024-09-02T12:06:07.828 INFO:tasks.nvmeof.[nvmeof.thrasher]:skipping daemon nvmeof.a with skip (0.6075858738729369) > weight (0.25) 2024-09-02T12:06:07.828 INFO:tasks.nvmeof.[nvmeof.thrasher]:skipping daemon nvmeof.b with skip (0.6280818374942808) > weight (0.25) 2024-09-02T12:06:07.828 INFO:tasks.nvmeof.[nvmeof.thrasher]:kill nvmeof.c 2024-09-02T12:06:07.829 DEBUG:teuthology.orchestra.run.smithi094:> ceph orch daemon stop nvmeof.nvmeof.c 2024-09-02T12:06:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:06:07.844 INFO:teuthology.orchestra.run.smithi094.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-02T12:06:07.844 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -- /home/ubuntu/cephtest/mnt.3 2024-09-02T12:06:07.899 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.3 2024-09-02T12:06:07.899 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/mnt.3 && mkdir -- client.3 2024-09-02T12:06:07.954 INFO:tasks.workunit:timeout=30m 2024-09-02T12:06:07.954 INFO:tasks.workunit:cleanup=True 2024-09-02T12:06:07.954 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/clone.client.3 && git clone https://github.com/vallariag/ceph.git /home/ubuntu/cephtest/clone.client.3 && cd /home/ubuntu/cephtest/clone.client.3 && git checkout 6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 2024-09-02T12:06:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:07 smithi087 ceph-mon[21100]: pgmap v375: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-02T12:06:08.010 INFO:tasks.workunit.client.3.smithi094.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.3'... 2024-09-02T12:06:08.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:07 smithi064 ceph-mon[17163]: pgmap v375: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-02T12:06:08.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:07 smithi094 ceph-mon[20404]: pgmap v375: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-02T12:06:08.137 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled to stop nvmeof.nvmeof.c on host 'smithi094' 2024-09-02T12:06:08.148 INFO:tasks.nvmeof.[nvmeof.thrasher]:skipping daemon nvmeof.d with skip (0.7365918592413352) > weight (0.25) 2024-09-02T12:06:08.148 INFO:tasks.nvmeof.[nvmeof.thrasher]:waiting for 109 secs before reviving 2024-09-02T12:06:09.263 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:08 smithi094 systemd[1]: Stopping Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:06:09.263 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "stop", "name": "nvmeof.nvmeof.c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:06:09.263 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: Schedule stop daemon nvmeof.nvmeof.c 2024-09-02T12:06:09.263 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.263 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.263 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:06:09.263 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"} : dispatch 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:06:09.264 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "stop", "name": "nvmeof.nvmeof.c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:06:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: Schedule stop daemon nvmeof.nvmeof.c 2024-09-02T12:06:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:06:09.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"} : dispatch 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:06:09.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:09 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "stop", "name": "nvmeof.nvmeof.c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:06:09.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: Schedule stop daemon nvmeof.nvmeof.c 2024-09-02T12:06:09.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"} : dispatch 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:06:09.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:09 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:09.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] ERROR server.py:111 (2): GatewayServer exception occurred: 2024-09-02T12:06:09.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: Traceback (most recent call last): 2024-09-02T12:06:09.596 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/src/control/__main__.py", line 39, in 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: gateway.keep_alive() 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/src/control/server.py", line 526, in keep_alive 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: timedout = self.server.wait_for_termination(timeout=1) 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/src/__pypackages__/3.9/lib/grpc/_server.py", line 1118, in wait_for_termination 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: return _common.wait(self._state.termination_event.wait, 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 157, in wait 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: _wait_once(wait_fn, remaining, spin_cb) 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 112, in _wait_once 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: wait_fn(timeout=timeout) 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/usr/lib64/python3.9/threading.py", line 581, in wait 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: signaled = self._cond.wait(timeout) 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/usr/lib64/python3.9/threading.py", line 316, in wait 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: gotit = waiter.acquire(True, timeout) 2024-09-02T12:06:09.597 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: File "/src/control/__main__.py", line 17, in sigterm_handler 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: raise SystemExit(f"Gateway process terminated") 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: SystemExit: Gateway process terminated 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.c) pid 19... 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: *** Caught signal (Aborted) ** 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: in thread 7f6dc46195c0 thread_name:ceph-nvmeof-mon 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: ceph version 19.0.0-4996-g0ec90b1e (0ec90b1e61a7489b13d6d8432156a0417f35db7f) squid (dev) 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6dc59ab6f0] 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 2: /lib64/libc.so.6(+0x8679a) [0x7f6dc59f379a] 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 3: pthread_cond_wait() 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f6dc5d436b0] 2024-09-02T12:06:09.598 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 5: (AsyncMessenger::wait()+0x7b) [0x7f6dc6832ceb] 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 6: (NVMeofGwMonitorClient::main(std::vector >)+0x18) [0x555cf9811b18] 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 7: main() 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 8: /lib64/libc.so.6(+0x29590) [0x7f6dc5996590] 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 9: __libc_start_main() 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: 10: _start() 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.c) pid 85... 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:125 (2): Stopping the server... 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:482 (2): Terminating discovery service... 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO state.py:545 (87): Cleanup OMAP on exit (discovery-smithi094) 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:489 (2): Discovery service terminated 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO state.py:545 (2): Cleanup OMAP on exit (gateway-client.nvmeof.nvmeof.c) 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:137 (2): Exiting the gateway process. 2024-09-02T12:06:09.599 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO utils.py:392 (2): Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvmeof.c/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvmeof.c/nvmeof-log.gz 2024-09-02T12:06:09.600 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 podman[46884]: 2024-09-02 12:06:09.312511765 +0000 UTC m=+0.229124695 container died 6c79565901db2f837e6b7d2aca3566abe4381be45c45a99d92c5109b7f167bb3 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.buildah.version=1.29.0, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.repo=https://github.com/ceph/spdk.git, url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.ceph-release=18.2.4, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, maintainer=Ceph Developers , io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.url=https://github.com/ceph/ceph-nvmeof, build-date=2024-07-30 15:47:38 UTC, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.spdk.build.target-arch=, summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, org.centos.version=9.0-21.el9, vendor=Red Hat, Inc., version=1.2.17, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.openshift.wants=ceph,rbd, release=, vcs-type=git, io.ceph.version=1.2.17, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.version=24.01, io.spdk.build.pkgdep-args=--rbd, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, architecture=x86_64, name=ceph-nvmeof, io.ceph.maintainer=Ceph Developers , io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.component=ceph-nvmeof, io.ceph.git.branch=tags/1.2.17, io.ceph.nvmeof=None, io.spdk.build.makeflags=, com.redhat.component=ubi9-container) 2024-09-02T12:06:10.022 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 podman[46884]: 2024-09-02 12:06:09.81134219 +0000 UTC m=+0.727955107 container cleanup 6c79565901db2f837e6b7d2aca3566abe4381be45c45a99d92c5109b7f167bb3 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.target-arch=, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.git.branch=tags/1.2.17, io.spdk.build.makeflags=, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., url=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.version=1.2.17, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.buildah.version=1.29.0, release=, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.summary=Ceph NVMe over Fabrics Gateway, summary=Ceph NVMe over Fabrics Gateway, io.ceph.component=ceph-nvmeof, org.centos.version=9.0-21.el9, name=ceph-nvmeof, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.openshift.wants=ceph,rbd, io.spdk.build.pkgdep-args=--rbd, io.ceph.nvmeof=None, version=1.2.17, io.spdk.build.ceph-release=18.2.4, maintainer=Ceph Developers , io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, build-date=2024-07-30 15:47:38 UTC, io.spdk.git.branch=undefined, com.redhat.component=ubi9-container, io.spdk.version=24.01, vendor=Red Hat, Inc., io.ceph.maintainer=Ceph Developers , distribution-scope=public, architecture=x86_64, io.k8s.display-name=Ceph NVMe over Fabrics Gateway) 2024-09-02T12:06:10.022 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:09 smithi094 bash[46884]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c 2024-09-02T12:06:10.346 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:10 smithi094 ceph-mon[20404]: pgmap v376: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-09-02T12:06:10.346 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:10 smithi094 podman[46896]: 2024-09-02 12:06:10.02130129 +0000 UTC m=+0.710585685 container remove 6c79565901db2f837e6b7d2aca3566abe4381be45c45a99d92c5109b7f167bb3 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, vendor=Red Hat, Inc., description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, url=https://github.com/ceph/ceph-nvmeof, io.spdk.version=24.01, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.buildah.version=1.29.0, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.summary=Ceph NVMe over Fabrics Gateway, version=1.2.17, io.ceph.component=ceph-nvmeof, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.openshift.wants=ceph,rbd, io.spdk.build.pkgdep-args=--rbd, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, name=ceph-nvmeof, architecture=x86_64, io.ceph.url=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, distribution-scope=public, com.redhat.component=ubi9-container, release=, io.ceph.git.branch=tags/1.2.17, io.ceph.version=1.2.17, io.spdk.build.makeflags=, io.spdk.build.target-arch=, maintainer=Ceph Developers , org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.nvmeof=None, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2024-07-30 15:47:38 UTC, org.centos.version=9.0-21.el9, io.ceph.maintainer=Ceph Developers , io.spdk.build.ceph-release=18.2.4) 2024-09-02T12:06:10.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:10 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Deactivated successfully. 2024-09-02T12:06:10.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:10 smithi094 systemd[1]: Stopped Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:06:10.347 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:06:10 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Consumed 5min 5.087s CPU time. 2024-09-02T12:06:10.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:10 smithi087 ceph-mon[21100]: pgmap v376: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-09-02T12:06:10.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:10 smithi064 ceph-mon[17163]: pgmap v376: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-09-02T12:06:11.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:11.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:11.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:11 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:06:11.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:11.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:11.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:11 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:06:11.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:11.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:11.753 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:11.753 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:11.753 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:11 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:06:11.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:11.845 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:11.916 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:11.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:12.021 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:12.094 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:12.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:12.196 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:12.227 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:12.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:12 smithi087 ceph-mon[21100]: pgmap v377: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-09-02T12:06:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:06:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:06:12.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:12 smithi064 ceph-mon[17163]: pgmap v377: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-09-02T12:06:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:06:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:06:12.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:12 smithi094 ceph-mon[20404]: pgmap v377: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-09-02T12:06:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:06:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:06:12.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:06:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:14 smithi064 ceph-mon[17163]: pgmap v378: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:06:14.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:14 smithi094 ceph-mon[20404]: pgmap v378: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:06:14.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:14 smithi087 ceph-mon[21100]: pgmap v378: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-09-02T12:06:16.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:16 smithi064 ceph-mon[17163]: pgmap v379: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-02T12:06:16.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:16 smithi094 ceph-mon[20404]: pgmap v379: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-02T12:06:16.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:16 smithi087 ceph-mon[21100]: pgmap v379: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-02T12:06:17.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:17.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:17.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:17.423 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:17.494 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:17.565 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:17.597 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:17.669 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:17.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:17.770 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:17.801 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:17.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:18.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:18 smithi064 ceph-mon[17163]: pgmap v380: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-02T12:06:18.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:18 smithi094 ceph-mon[20404]: pgmap v380: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-02T12:06:18.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:18 smithi087 ceph-mon[21100]: pgmap v380: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-02T12:06:20.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:20 smithi064 ceph-mon[17163]: pgmap v381: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2 op/s 2024-09-02T12:06:20.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:20 smithi094 ceph-mon[20404]: pgmap v381: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2 op/s 2024-09-02T12:06:20.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:20 smithi087 ceph-mon[21100]: pgmap v381: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 2 op/s 2024-09-02T12:06:22.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:22 smithi064 ceph-mon[17163]: pgmap v382: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3 op/s 2024-09-02T12:06:22.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:22 smithi094 ceph-mon[20404]: pgmap v382: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3 op/s 2024-09-02T12:06:22.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:22 smithi087 ceph-mon[21100]: pgmap v382: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3 op/s 2024-09-02T12:06:22.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:22.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:22.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:22.995 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:23.066 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:23.138 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:23.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:23.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:23.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:06:23] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:06:23.345 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:06:23.346 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:06:23.346 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:06:23.346 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:06:23.346 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:37516 2024-09-02T12:06:23.346 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:06:23.017419] bdev_rbd.c:1039:_bdev_rbd_wait_for_latest_osdmap: *NOTICE*: rados_wait_for_latest_osdmap cluster: cluster_context_3_0 elapsed time: 0.000292 seconds 2024-09-02T12:06:23.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:23.388 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:23.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:24.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:24 smithi064 ceph-mon[17163]: pgmap v383: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:24.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:24 smithi094 ceph-mon[20404]: pgmap v383: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:24 smithi087 ceph-mon[21100]: pgmap v383: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:26.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:26 smithi064 ceph-mon[17163]: pgmap v384: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:26.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:26 smithi094 ceph-mon[20404]: pgmap v384: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:26 smithi087 ceph-mon[21100]: pgmap v384: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:28.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:28.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:28.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:28.574 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:28 smithi087 ceph-mon[21100]: pgmap v385: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:28.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:28.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:28 smithi064 ceph-mon[17163]: pgmap v385: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:28.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:28 smithi094 ceph-mon[20404]: pgmap v385: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:28.656 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:28.727 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:28.775 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:28.808 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:28.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:28.920 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:28.951 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:28.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:30.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:30 smithi064 ceph-mon[17163]: pgmap v386: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:30.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:30 smithi094 ceph-mon[20404]: pgmap v386: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:30.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:30 smithi087 ceph-mon[21100]: pgmap v386: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:32.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:32 smithi064 ceph-mon[17163]: pgmap v387: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:32.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:32 smithi094 ceph-mon[20404]: pgmap v387: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:32.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:32 smithi087 ceph-mon[21100]: pgmap v387: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:34.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:34.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:34.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:34.160 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:34.231 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:34.302 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:34.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:34.422 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:34.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:34.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:34 smithi087 ceph-mon[21100]: pgmap v388: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:34.527 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:34.558 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:34.579 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:34 smithi094 ceph-mon[20404]: pgmap v388: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:34.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:34.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:34 smithi064 ceph-mon[17163]: pgmap v388: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:36.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:36 smithi064 ceph-mon[17163]: pgmap v389: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:36.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:36 smithi094 ceph-mon[20404]: pgmap v389: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:36.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:36 smithi087 ceph-mon[21100]: pgmap v389: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:38.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:38 smithi064 ceph-mon[17163]: pgmap v390: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:38.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:38 smithi094 ceph-mon[20404]: pgmap v390: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:38.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:38 smithi087 ceph-mon[21100]: pgmap v390: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:39.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:39.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:39.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:39.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:39.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:39.897 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:39.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:40.005 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:40.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:40.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:40.144 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:40.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:40.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:40 smithi087 ceph-mon[21100]: pgmap v391: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:40.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:40 smithi064 ceph-mon[17163]: pgmap v391: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:40.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:40 smithi094 ceph-mon[20404]: pgmap v391: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:42.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:42 smithi064 ceph-mon[17163]: pgmap v392: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:42.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:42 smithi094 ceph-mon[20404]: pgmap v392: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:42.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:42 smithi087 ceph-mon[21100]: pgmap v392: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:44.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:44 smithi064 ceph-mon[17163]: pgmap v393: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:44.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:44 smithi094 ceph-mon[20404]: pgmap v393: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:44.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:44 smithi087 ceph-mon[21100]: pgmap v393: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:45.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:45.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:45.308 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:45.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:45.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:45.483 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:45.516 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:45.588 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:45.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:45.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:45.722 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:45.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:46.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:46 smithi064 ceph-mon[17163]: pgmap v394: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:46.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:46 smithi094 ceph-mon[20404]: pgmap v394: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:46.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:46 smithi087 ceph-mon[21100]: pgmap v394: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:48.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:48 smithi064 ceph-mon[17163]: pgmap v395: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:48.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:48 smithi094 ceph-mon[20404]: pgmap v395: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:48.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:48 smithi087 ceph-mon[21100]: pgmap v395: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:49.094 INFO:tasks.workunit.client.3.smithi094.stderr:Note: switching to '6bdd3521695a0621868ce5034eaba1fd0b6e3cd1'. 2024-09-02T12:06:49.094 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr:state without impacting any branches by switching back to a branch. 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr: git switch -c 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr:Or undo this operation with: 2024-09-02T12:06:49.095 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.096 INFO:tasks.workunit.client.3.smithi094.stderr: git switch - 2024-09-02T12:06:49.096 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.096 INFO:tasks.workunit.client.3.smithi094.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-02T12:06:49.096 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:06:49.096 INFO:tasks.workunit.client.3.smithi094.stderr:HEAD is now at 6bdd3521695 qa/suites/nvmeof: wait for service "nvmeof.mypool.mygroup0" 2024-09-02T12:06:49.098 DEBUG:teuthology.orchestra.run.smithi094:> cd -- /home/ubuntu/cephtest/clone.client.3/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.3 2024-09-02T12:06:49.204 INFO:tasks.workunit.client.3.smithi094.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-02T12:06:49.206 INFO:tasks.workunit.client.3.smithi094.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.3/qa/workunits/direct_io' 2024-09-02T12:06:49.206 INFO:tasks.workunit.client.3.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-02T12:06:49.827 INFO:tasks.workunit.client.3.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-02T12:06:49.886 INFO:tasks.workunit.client.3.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-02T12:06:49.925 INFO:tasks.workunit.client.3.smithi094.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.3/qa/workunits/direct_io' 2024-09-02T12:06:49.927 INFO:tasks.workunit.client.3.smithi094.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.3/qa/workunits/fs' 2024-09-02T12:06:49.927 INFO:tasks.workunit.client.3.smithi094.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-02T12:06:49.965 INFO:tasks.workunit.client.3.smithi094.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.3/qa/workunits/fs' 2024-09-02T12:06:49.968 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2024-09-02T12:06:49.969 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/home/ubuntu/cephtest/workunits.list.client.3 of=/dev/stdout 2024-09-02T12:06:49.984 INFO:tasks.workunit:Running workunits matching nvmeof/fio_test.sh on client.3... 2024-09-02T12:06:49.985 INFO:tasks.workunit:Running workunit nvmeof/fio_test.sh... 2024-09-02T12:06:49.985 DEBUG:teuthology.orchestra.run.smithi094:workunit test nvmeof/fio_test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 IOSTAT_INTERVAL=10 RBD_POOL=mypool RUNTIME=600 timeout 30m /home/ubuntu/cephtest/clone.client.3/qa/workunits/nvmeof/fio_test.sh --rbd_iostat 2024-09-02T12:06:50.040 INFO:tasks.workunit.client.3.smithi094.stderr:+ sudo yum -y install fio 2024-09-02T12:06:50.583 INFO:tasks.workunit.client.3.smithi094.stdout:Last metadata expiration check: 0:16:06 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T12:06:50.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:50 smithi064 ceph-mon[17163]: pgmap v396: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:50.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:50 smithi094 ceph-mon[20404]: pgmap v396: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:50.716 INFO:tasks.workunit.client.3.smithi094.stdout:Dependencies resolved. 2024-09-02T12:06:50.717 INFO:tasks.workunit.client.3.smithi094.stdout:================================================================================ 2024-09-02T12:06:50.717 INFO:tasks.workunit.client.3.smithi094.stdout: Package Architecture Version Repository Size 2024-09-02T12:06:50.717 INFO:tasks.workunit.client.3.smithi094.stdout:================================================================================ 2024-09-02T12:06:50.717 INFO:tasks.workunit.client.3.smithi094.stdout:Installing: 2024-09-02T12:06:50.717 INFO:tasks.workunit.client.3.smithi094.stdout: fio x86_64 3.35-1.el9 appstream 6.2 M 2024-09-02T12:06:50.717 INFO:tasks.workunit.client.3.smithi094.stdout:Installing dependencies: 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout:Installing weak dependencies: 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-http x86_64 3.35-1.el9 appstream 18 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-libaio x86_64 3.35-1.el9 appstream 16 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-nbd x86_64 3.35-1.el9 appstream 15 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-rados x86_64 3.35-1.el9 appstream 16 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-rbd x86_64 3.35-1.el9 appstream 16 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-rdma x86_64 3.35-1.el9 appstream 21 k 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout:Transaction Summary 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout:================================================================================ 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout:Install 8 Packages 2024-09-02T12:06:50.718 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:06:50.719 INFO:tasks.workunit.client.3.smithi094.stdout:Total download size: 6.5 M 2024-09-02T12:06:50.719 INFO:tasks.workunit.client.3.smithi094.stdout:Installed size: 9.1 M 2024-09-02T12:06:50.719 INFO:tasks.workunit.client.3.smithi094.stdout:Downloading Packages: 2024-09-02T12:06:50.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:50 smithi087 ceph-mon[21100]: pgmap v396: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:50.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:50.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:50.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:50.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:50.990 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:51.061 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:51.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:51.167 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:51.220 INFO:tasks.workunit.client.3.smithi094.stdout:(1/8): fio-engine-http-3.35-1.el9.x86_64.rpm 45 kB/s | 18 kB 00:00 2024-09-02T12:06:51.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:51.244 INFO:tasks.workunit.client.3.smithi094.stdout:(2/8): fio-engine-libaio-3.35-1.el9.x86_64.rpm 37 kB/s | 16 kB 00:00 2024-09-02T12:06:51.269 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:51.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:51.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:51.353 INFO:tasks.workunit.client.3.smithi094.stdout:(3/8): fio-engine-rados-3.35-1.el9.x86_64.rpm 145 kB/s | 16 kB 00:00 2024-09-02T12:06:51.378 INFO:tasks.workunit.client.3.smithi094.stdout:(4/8): fio-engine-nbd-3.35-1.el9.x86_64.rpm 109 kB/s | 15 kB 00:00 2024-09-02T12:06:51.454 INFO:tasks.workunit.client.3.smithi094.stdout:(5/8): fio-engine-rbd-3.35-1.el9.x86_64.rpm 162 kB/s | 16 kB 00:00 2024-09-02T12:06:51.480 INFO:tasks.workunit.client.3.smithi094.stdout:(6/8): fio-engine-rdma-3.35-1.el9.x86_64.rpm 205 kB/s | 21 kB 00:00 2024-09-02T12:06:51.821 INFO:tasks.workunit.client.3.smithi094.stdout:(7/8): libnbd-1.20.2-2.el9.x86_64.rpm 465 kB/s | 171 kB 00:00 2024-09-02T12:06:52.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:52 smithi064 ceph-mon[17163]: pgmap v397: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:52.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:52 smithi094 ceph-mon[20404]: pgmap v397: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:52.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:52 smithi087 ceph-mon[21100]: pgmap v397: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:54.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:54 smithi064 ceph-mon[17163]: pgmap v398: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:54.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:54 smithi094 ceph-mon[20404]: pgmap v398: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:54.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:54 smithi087 ceph-mon[21100]: pgmap v398: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:55.387 INFO:tasks.workunit.client.3.smithi094.stdout:(8/8): fio-3.35-1.el9.x86_64.rpm 1.4 MB/s | 6.2 MB 00:04 2024-09-02T12:06:55.387 INFO:tasks.workunit.client.3.smithi094.stdout:-------------------------------------------------------------------------------- 2024-09-02T12:06:55.387 INFO:tasks.workunit.client.3.smithi094.stdout:Total 1.4 MB/s | 6.5 MB 00:04 2024-09-02T12:06:55.523 INFO:tasks.workunit.client.3.smithi094.stdout:Running transaction check 2024-09-02T12:06:55.536 INFO:tasks.workunit.client.3.smithi094.stdout:Transaction check succeeded. 2024-09-02T12:06:55.536 INFO:tasks.workunit.client.3.smithi094.stdout:Running transaction test 2024-09-02T12:06:55.665 INFO:tasks.workunit.client.3.smithi094.stdout:Transaction test succeeded. 2024-09-02T12:06:55.666 INFO:tasks.workunit.client.3.smithi094.stdout:Running transaction 2024-09-02T12:06:56.052 INFO:tasks.workunit.client.3.smithi094.stdout: Preparing : 1/1 2024-09-02T12:06:56.116 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 1/8 2024-09-02T12:06:56.141 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-engine-http-3.35-1.el9.x86_64 2/8 2024-09-02T12:06:56.166 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-engine-libaio-3.35-1.el9.x86_64 3/8 2024-09-02T12:06:56.192 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-engine-nbd-3.35-1.el9.x86_64 4/8 2024-09-02T12:06:56.217 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-engine-rados-3.35-1.el9.x86_64 5/8 2024-09-02T12:06:56.242 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-engine-rbd-3.35-1.el9.x86_64 6/8 2024-09-02T12:06:56.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:06:56.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:06:56.396 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-engine-rdma-3.35-1.el9.x86_64 7/8 2024-09-02T12:06:56.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:06:56.481 INFO:tasks.workunit.client.3.smithi094.stdout: Installing : fio-3.35-1.el9.x86_64 8/8 2024-09-02T12:06:56.493 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:06:56.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:06:56.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:56 smithi064 ceph-mon[17163]: pgmap v399: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:56.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:56 smithi094 ceph-mon[20404]: pgmap v399: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:56.628 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:56 smithi087 ceph-mon[21100]: pgmap v399: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:06:56.638 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:06:56.673 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:06:56.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:06:56.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:06:56.845 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:06:56.875 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:06:56.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:06:58.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:06:58 smithi064 ceph-mon[17163]: pgmap v400: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:58.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:06:58 smithi094 ceph-mon[20404]: pgmap v400: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:58.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:06:58 smithi087 ceph-mon[21100]: pgmap v400: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:06:59.070 INFO:tasks.workunit.client.3.smithi094.stdout: Running scriptlet: fio-3.35-1.el9.x86_64 8/8 2024-09-02T12:06:59.070 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-3.35-1.el9.x86_64 1/8 2024-09-02T12:06:59.070 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-engine-http-3.35-1.el9.x86_64 2/8 2024-09-02T12:06:59.071 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-engine-libaio-3.35-1.el9.x86_64 3/8 2024-09-02T12:06:59.071 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-engine-nbd-3.35-1.el9.x86_64 4/8 2024-09-02T12:06:59.071 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-engine-rados-3.35-1.el9.x86_64 5/8 2024-09-02T12:06:59.071 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-engine-rbd-3.35-1.el9.x86_64 6/8 2024-09-02T12:06:59.071 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : fio-engine-rdma-3.35-1.el9.x86_64 7/8 2024-09-02T12:06:59.484 INFO:tasks.workunit.client.3.smithi094.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 8/8 2024-09-02T12:06:59.484 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:06:59.484 INFO:tasks.workunit.client.3.smithi094.stdout:Installed: 2024-09-02T12:06:59.484 INFO:tasks.workunit.client.3.smithi094.stdout: fio-3.35-1.el9.x86_64 fio-engine-http-3.35-1.el9.x86_64 2024-09-02T12:06:59.485 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-libaio-3.35-1.el9.x86_64 fio-engine-nbd-3.35-1.el9.x86_64 2024-09-02T12:06:59.485 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-rados-3.35-1.el9.x86_64 fio-engine-rbd-3.35-1.el9.x86_64 2024-09-02T12:06:59.485 INFO:tasks.workunit.client.3.smithi094.stdout: fio-engine-rdma-3.35-1.el9.x86_64 libnbd-1.20.2-2.el9.x86_64 2024-09-02T12:06:59.485 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:06:59.485 INFO:tasks.workunit.client.3.smithi094.stdout:Complete! 2024-09-02T12:06:59.812 INFO:tasks.workunit.client.3.smithi094.stderr:+ sudo yum -y install sysstat 2024-09-02T12:07:00.372 INFO:tasks.workunit.client.3.smithi094.stdout:Last metadata expiration check: 0:16:16 ago on Mon 02 Sep 2024 11:50:44 AM UTC. 2024-09-02T12:07:00.460 INFO:tasks.workunit.client.3.smithi094.stdout:Package sysstat-12.5.4-9.el9.x86_64 is already installed. 2024-09-02T12:07:00.504 INFO:tasks.workunit.client.3.smithi094.stdout:Dependencies resolved. 2024-09-02T12:07:00.505 INFO:tasks.workunit.client.3.smithi094.stdout:Nothing to do. 2024-09-02T12:07:00.505 INFO:tasks.workunit.client.3.smithi094.stdout:Complete! 2024-09-02T12:07:00.578 INFO:tasks.workunit.client.3.smithi094.stderr:+ namespace_range_start= 2024-09-02T12:07:00.578 INFO:tasks.workunit.client.3.smithi094.stderr:+ namespace_range_end= 2024-09-02T12:07:00.578 INFO:tasks.workunit.client.3.smithi094.stderr:+ rbd_iostat=false 2024-09-02T12:07:00.578 INFO:tasks.workunit.client.3.smithi094.stderr:+ [[ 1 -gt 0 ]] 2024-09-02T12:07:00.579 INFO:tasks.workunit.client.3.smithi094.stderr:+ case "$1" in 2024-09-02T12:07:00.579 INFO:tasks.workunit.client.3.smithi094.stderr:+ rbd_iostat=true 2024-09-02T12:07:00.579 INFO:tasks.workunit.client.3.smithi094.stderr:+ shift 2024-09-02T12:07:00.579 INFO:tasks.workunit.client.3.smithi094.stderr:+ [[ 0 -gt 0 ]] 2024-09-02T12:07:00.579 INFO:tasks.workunit.client.3.smithi094.stderr:++ mktemp -t nvmeof-fio-XXXX 2024-09-02T12:07:00.579 INFO:tasks.workunit.client.3.smithi094.stderr:+ fio_file=/tmp/nvmeof-fio-1GLI 2024-09-02T12:07:00.580 INFO:tasks.workunit.client.3.smithi094.stderr:++ sudo nvme list --output-format=json 2024-09-02T12:07:00.580 INFO:tasks.workunit.client.3.smithi094.stderr:++ jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == "Ceph bdev Controller") | .DevicePath' 2024-09-02T12:07:00.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:00 smithi094 ceph-mon[20404]: pgmap v401: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:00.603 INFO:tasks.workunit.client.3.smithi094.stderr:+ all_drives_list= 2024-09-02T12:07:00.603 INFO:tasks.workunit.client.3.smithi094.stderr:+ '[' '' ']' 2024-09-02T12:07:00.603 INFO:tasks.workunit.client.3.smithi094.stderr:+ '[' '' ']' 2024-09-02T12:07:00.603 INFO:tasks.workunit.client.3.smithi094.stderr:+ selected_drives= 2024-09-02T12:07:00.603 INFO:tasks.workunit.client.3.smithi094.stderr:+ RUNTIME=600 2024-09-02T12:07:00.604 INFO:tasks.workunit.client.3.smithi094.stderr:+ cat 2024-09-02T12:07:00.604 INFO:tasks.workunit.client.3.smithi094.stderr:++ echo '' 2024-09-02T12:07:00.604 INFO:tasks.workunit.client.3.smithi094.stderr:++ tr '\n' : 2024-09-02T12:07:00.604 INFO:tasks.workunit.client.3.smithi094.stderr:++ sed 's/:$//' 2024-09-02T12:07:00.605 INFO:tasks.workunit.client.3.smithi094.stderr:+ echo '[nvmeof.fio] starting fio test...' 2024-09-02T12:07:00.608 INFO:tasks.workunit.client.3.smithi094.stdout:[nvmeof.fio] starting fio test... 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ '[' -n 10 ']' 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ iostat_count=60 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ '[' true = true ']' 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ iterations=120 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ iostat -d -p 10 60 -h 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ fio --showcmd /tmp/nvmeof-fio-1GLI 2024-09-02T12:07:00.609 INFO:tasks.workunit.client.3.smithi094.stderr:+ timeout 20 rbd perf image iostat mypool --iterations 120 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout:Linux 5.14.0-503.el9.x86_64 (smithi094) 09/02/2024 _x86_64_ (8 CPU) 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 30.96 151.5k 955.1k 12.7M 178.2M 1.1G 14.9G nvme0n1 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 46.19 411.7k 5.9M 0.0k 484.0M 6.9G 0.0k sda 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 46.10 409.2k 5.9M 0.0k 481.1M 6.9G 0.0k sda1 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.40 8.1k 0.0k 0.0k 9.5M 4.0k 0.0k dm-0 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 5.72 42.1k 15.6k 0.0k 49.5M 18.4M 0.0k dm-1 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 14.45 37.9k 370.4k 0.0k 44.5M 435.5M 0.0k dm-2 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 14.67 36.2k 372.1k 0.0k 42.5M 437.4M 0.0k dm-3 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.97 4.6k 102.3k 12.7M 5.4M 120.3M 14.9G dm-4 2024-09-02T12:07:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:00.613 INFO:tasks.workunit.client.3.smithi094.stderr:fio: failed parsing filename= 2024-09-02T12:07:00.613 INFO:tasks.workunit.client.3.smithi094.stderr:fio: job nvmeof-fio-test dropped 2024-09-02T12:07:00.614 INFO:tasks.workunit.client.3.smithi094.stdout:fio --name=nvmeof-fio-test 2024-09-02T12:07:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:07:00.642 INFO:tasks.workunit.client.3.smithi094.stderr:rbd: waiting for initial image stats 2024-09-02T12:07:00.642 INFO:tasks.workunit.client.3.smithi094.stderr: 2024-09-02T12:07:00.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:00 smithi087 ceph-mon[21100]: pgmap v401: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:00.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:00 smithi064 ceph-mon[17163]: pgmap v401: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:01.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:01.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:02.038 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:02.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:02.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:02.216 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:02.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:02.334 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:02.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:02.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:02.473 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:02.496 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: pgmap v402: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: { 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: "prefix": "rbd perf image stats", 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: "pool_spec": "mypool", 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: "sort_by": "write_ops", 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: "format": "json" 2024-09-02T12:07:02.497 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:02 smithi094 ceph-mon[20404]: }"]: dispatch 2024-09-02T12:07:02.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: pgmap v402: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: { 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: "prefix": "rbd perf image stats", 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: "pool_spec": "mypool", 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: "sort_by": "write_ops", 2024-09-02T12:07:02.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: "format": "json" 2024-09-02T12:07:02.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:02 smithi087 ceph-mon[21100]: }"]: dispatch 2024-09-02T12:07:02.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: pgmap v402: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:02.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:02.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: { 2024-09-02T12:07:02.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: "prefix": "rbd perf image stats", 2024-09-02T12:07:02.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: "pool_spec": "mypool", 2024-09-02T12:07:02.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: "sort_by": "write_ops", 2024-09-02T12:07:02.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: "format": "json" 2024-09-02T12:07:02.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:02 smithi064 ceph-mon[17163]: }"]: dispatch 2024-09-02T12:07:04.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:04 smithi087 ceph-mon[21100]: pgmap v403: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:04.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:04 smithi064 ceph-mon[17163]: pgmap v403: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:04.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:04 smithi094 ceph-mon[20404]: pgmap v403: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:06.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:06 smithi087 ceph-mon[21100]: pgmap v404: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:06.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:06 smithi064 ceph-mon[17163]: pgmap v404: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:06.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:06 smithi094 ceph-mon[20404]: pgmap v404: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:07.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:07.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:07.596 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:07.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: { 2024-09-02T12:07:07.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: "prefix": "rbd perf image stats", 2024-09-02T12:07:07.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: "pool_spec": "mypool", 2024-09-02T12:07:07.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: "sort_by": "write_ops", 2024-09-02T12:07:07.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: "format": "json" 2024-09-02T12:07:07.597 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:07 smithi094 ceph-mon[20404]: }"]: dispatch 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: { 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: "prefix": "rbd perf image stats", 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: "pool_spec": "mypool", 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: "sort_by": "write_ops", 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: "format": "json" 2024-09-02T12:07:07.630 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:07 smithi064 ceph-mon[17163]: }"]: dispatch 2024-09-02T12:07:07.639 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: { 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: "prefix": "rbd perf image stats", 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: "pool_spec": "mypool", 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: "sort_by": "write_ops", 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: "format": "json" 2024-09-02T12:07:07.661 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:07 smithi087 ceph-mon[21100]: }"]: dispatch 2024-09-02T12:07:07.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:07.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:07.817 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:07.852 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:07.925 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:07.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:08.028 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:08.060 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:08.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:08.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:08 smithi087 ceph-mon[21100]: pgmap v405: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:08.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:08 smithi064 ceph-mon[17163]: pgmap v405: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:08.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:08 smithi094 ceph-mon[20404]: pgmap v405: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:10.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 8.00 0.0k 26.2k 0.0k 0.0k 262.0k 0.0k nvme0n1 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 3.80 0.4k 850.8k 0.0k 4.0k 8.3M 0.0k sda 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 3.80 0.4k 850.8k 0.0k 4.0k 8.3M 0.0k sda1 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 8.00 0.0k 26.2k 0.0k 0.0k 262.0k 0.0k dm-4 2024-09-02T12:07:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:10.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:10 smithi087 ceph-mon[21100]: pgmap v406: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:10.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:10 smithi064 ceph-mon[17163]: pgmap v406: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:10.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:10 smithi094 ceph-mon[20404]: pgmap v406: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: pgmap v407: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: { 2024-09-02T12:07:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: "prefix": "rbd perf image stats", 2024-09-02T12:07:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: "pool_spec": "mypool", 2024-09-02T12:07:12.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: "sort_by": "write_ops", 2024-09-02T12:07:12.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: "format": "json" 2024-09-02T12:07:12.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: }"]: dispatch 2024-09-02T12:07:12.751 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:12 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:07:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: pgmap v407: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: { 2024-09-02T12:07:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: "prefix": "rbd perf image stats", 2024-09-02T12:07:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: "pool_spec": "mypool", 2024-09-02T12:07:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: "sort_by": "write_ops", 2024-09-02T12:07:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: "format": "json" 2024-09-02T12:07:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: }"]: dispatch 2024-09-02T12:07:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:12 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:07:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: pgmap v407: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: { 2024-09-02T12:07:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: "prefix": "rbd perf image stats", 2024-09-02T12:07:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: "pool_spec": "mypool", 2024-09-02T12:07:12.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: "sort_by": "write_ops", 2024-09-02T12:07:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: "format": "json" 2024-09-02T12:07:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: }"]: dispatch 2024-09-02T12:07:12.847 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:12 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:07:13.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:13.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:13.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:13.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:13.326 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:13.400 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:13.437 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:13 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T12:07:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:13 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T12:07:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:13 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:07:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:13 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:07:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:13 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:07:13.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:13 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:07:13.511 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:13.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:13 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T12:07:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:13 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T12:07:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:13 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:07:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:13 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:07:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:13 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:07:13.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:13 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:07:13.615 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:13.648 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:13.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:13 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T12:07:13.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:13 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T12:07:13.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:13 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:07:13.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:13 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:07:13.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:13 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:07:13.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:13 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:07:13.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:14.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:14 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi064 to 6496M 2024-09-02T12:07:14.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:14 smithi087 ceph-mon[21100]: pgmap v408: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:14 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi064 to 6496M 2024-09-02T12:07:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:14 smithi064 ceph-mon[17163]: pgmap v408: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:14.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:14 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi064 to 6496M 2024-09-02T12:07:14.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:14 smithi094 ceph-mon[20404]: pgmap v408: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:16.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:16 smithi087 ceph-mon[21100]: pgmap v409: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:16.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:16 smithi064 ceph-mon[17163]: pgmap v409: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:16.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:16 smithi094 ceph-mon[20404]: pgmap v409: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: { 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: "prefix": "rbd perf image stats", 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: "pool_spec": "mypool", 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: "sort_by": "write_ops", 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: "format": "json" 2024-09-02T12:07:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:17 smithi087 ceph-mon[21100]: }"]: dispatch 2024-09-02T12:07:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: { 2024-09-02T12:07:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: "prefix": "rbd perf image stats", 2024-09-02T12:07:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: "pool_spec": "mypool", 2024-09-02T12:07:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: "sort_by": "write_ops", 2024-09-02T12:07:17.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: "format": "json" 2024-09-02T12:07:17.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:17 smithi064 ceph-mon[17163]: }"]: dispatch 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: from='client.24949 -' entity='client.admin' cmd=[ 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: { 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: "prefix": "rbd perf image stats", 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: "pool_spec": "mypool", 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: "sort_by": "write_ops", 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: "format": "json" 2024-09-02T12:07:17.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:17 smithi094 ceph-mon[20404]: }"]: dispatch 2024-09-02T12:07:18.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:18.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:18 smithi064 ceph-mon[17163]: pgmap v410: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:18.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:18.753 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:18 smithi087 ceph-mon[21100]: pgmap v410: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:18.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:18.846 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:18.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:18 smithi094 ceph-mon[20404]: pgmap v410: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:18.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:18.992 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:19.030 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:19.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:19.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:19.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:19.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:19.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:20.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:20.610 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:07:20.610 INFO:tasks.workunit.client.3.smithi094.stdout: 10.20 0.0k 34.6k 0.0k 0.0k 346.5k 0.0k nvme0n1 2024-09-02T12:07:20.610 INFO:tasks.workunit.client.3.smithi094.stdout: 3.80 0.0k 824.4k 0.0k 0.0k 8.1M 0.0k sda 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 3.80 0.0k 824.4k 0.0k 0.0k 8.1M 0.0k sda1 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 10.30 0.0k 34.6k 0.0k 0.0k 346.5k 0.0k dm-4 2024-09-02T12:07:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:20.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:20 smithi087 ceph-mon[21100]: pgmap v411: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:20.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:20 smithi064 ceph-mon[17163]: pgmap v411: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:20.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:20 smithi094 ceph-mon[20404]: pgmap v411: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:22.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:22 smithi087 ceph-mon[21100]: pgmap v412: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:22.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:22 smithi064 ceph-mon[17163]: pgmap v412: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:22.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:22 smithi094 ceph-mon[20404]: pgmap v412: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:24.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:24.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:24.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:24.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:24.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:24.584 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:24.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:24 smithi064 ceph-mon[17163]: pgmap v413: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:24.622 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:24.687 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:24 smithi094 ceph-mon[20404]: pgmap v413: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:24.698 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:24.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:24 smithi087 ceph-mon[21100]: pgmap v413: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:24.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:24.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:24.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:24.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:26.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:26 smithi087 ceph-mon[21100]: pgmap v414: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:26.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:26 smithi064 ceph-mon[17163]: pgmap v414: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:26.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:26 smithi094 ceph-mon[20404]: pgmap v414: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:28.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:28 smithi087 ceph-mon[21100]: pgmap v415: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:28.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:28 smithi064 ceph-mon[17163]: pgmap v415: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:28.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:28 smithi094 ceph-mon[20404]: pgmap v415: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:29.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:29.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:30.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:30.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:30.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:30.175 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:30.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:30.290 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:30.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:30.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:30.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:30.457 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:30 smithi094 ceph-mon[20404]: pgmap v416: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:30.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:30.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:30.610 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:07:30.610 INFO:tasks.workunit.client.3.smithi094.stdout: 7.10 0.0k 23.3k 0.0k 0.0k 233.0k 0.0k nvme0n1 2024-09-02T12:07:30.610 INFO:tasks.workunit.client.3.smithi094.stdout: 4.10 0.0k 1.1M 0.0k 0.0k 10.6M 0.0k sda 2024-09-02T12:07:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 4.10 0.0k 1.1M 0.0k 0.0k 10.6M 0.0k sda1 2024-09-02T12:07:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:07:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:07:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:07:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:07:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.10 0.0k 23.3k 0.0k 0.0k 233.0k 0.0k dm-4 2024-09-02T12:07:30.612 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:30.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:30 smithi087 ceph-mon[21100]: pgmap v416: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:30.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:30 smithi064 ceph-mon[17163]: pgmap v416: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:32.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:32 smithi087 ceph-mon[21100]: pgmap v417: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:32.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:32 smithi064 ceph-mon[17163]: pgmap v417: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:32.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:32 smithi094 ceph-mon[20404]: pgmap v417: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:34.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:34 smithi087 ceph-mon[21100]: pgmap v418: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:34.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:34 smithi064 ceph-mon[17163]: pgmap v418: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:34.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:34 smithi094 ceph-mon[20404]: pgmap v418: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:35.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:35.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:35.591 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:35.622 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:35.693 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:35.764 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:35.801 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:35.876 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:35.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:35.982 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:36.015 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:36.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:36.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:36 smithi087 ceph-mon[21100]: pgmap v419: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:36.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:36 smithi064 ceph-mon[17163]: pgmap v419: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:36.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:36 smithi094 ceph-mon[20404]: pgmap v419: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:38.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:38 smithi087 ceph-mon[21100]: pgmap v420: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:38.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:38 smithi064 ceph-mon[17163]: pgmap v420: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:38.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:38 smithi094 ceph-mon[20404]: pgmap v420: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:39.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:39 smithi087 ceph-mon[21100]: pgmap v421: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:39.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:39 smithi064 ceph-mon[17163]: pgmap v421: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:39.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:39 smithi094 ceph-mon[20404]: pgmap v421: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:40.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:40.610 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.90 0.0k 22.4k 0.0k 0.0k 224.0k 0.0k nvme0n1 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 10.80 0.0k 745.6k 0.0k 0.0k 7.3M 0.0k sda 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 10.80 0.0k 745.6k 0.0k 0.0k 7.3M 0.0k sda1 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.90 0.0k 22.4k 0.0k 0.0k 224.0k 0.0k dm-4 2024-09-02T12:07:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:41.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:41.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:41.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:41.230 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:41.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:41.390 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:41.431 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:41.508 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:41.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:41.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:41.647 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:41.670 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:41 smithi094 ceph-mon[20404]: pgmap v422: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:41.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:42.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:41 smithi087 ceph-mon[21100]: pgmap v422: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:42.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:41 smithi064 ceph-mon[17163]: pgmap v422: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:44.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:43 smithi087 ceph-mon[21100]: pgmap v423: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:44.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:43 smithi064 ceph-mon[17163]: pgmap v423: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:44.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:43 smithi094 ceph-mon[20404]: pgmap v423: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:45.895 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:45 smithi064 ceph-mon[17163]: pgmap v424: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:46.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:45 smithi087 ceph-mon[21100]: pgmap v424: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:46.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:45 smithi094 ceph-mon[20404]: pgmap v424: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:46.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:46.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:46.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:46.846 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:46.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:46.991 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:47.029 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:47.105 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:47.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:47.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:47.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:47.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:48.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:47 smithi087 ceph-mon[21100]: pgmap v425: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:48.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:47 smithi064 ceph-mon[17163]: pgmap v425: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:47 smithi094 ceph-mon[20404]: pgmap v425: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:50.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:49 smithi087 ceph-mon[21100]: pgmap v426: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:50.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:49 smithi064 ceph-mon[17163]: pgmap v426: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:50.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:49 smithi094 ceph-mon[20404]: pgmap v426: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:50.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.80 0.0k 27.4k 0.0k 0.0k 273.5k 0.0k nvme0n1 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 3.50 0.0k 926.4k 0.0k 0.0k 9.0M 0.0k sda 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 3.50 0.0k 926.4k 0.0k 0.0k 9.0M 0.0k sda1 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 8.10 0.0k 27.4k 0.0k 0.0k 273.5k 0.0k dm-4 2024-09-02T12:07:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:07:52.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:51 smithi087 ceph-mon[21100]: pgmap v427: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:52.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:51 smithi064 ceph-mon[17163]: pgmap v427: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:52.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:51 smithi094 ceph-mon[20404]: pgmap v427: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:52.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:52.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:52.406 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:52.439 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:52.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:52.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:52.624 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:52.699 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:52.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:52.803 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:52.836 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:52.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:54.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:53 smithi087 ceph-mon[21100]: pgmap v428: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:54.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:53 smithi064 ceph-mon[17163]: pgmap v428: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:54.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:53 smithi094 ceph-mon[20404]: pgmap v428: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:55 smithi087 ceph-mon[21100]: pgmap v429: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:56.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:55 smithi064 ceph-mon[17163]: pgmap v429: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:56.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:55 smithi094 ceph-mon[20404]: pgmap v429: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:07:57.149 INFO:tasks.nvmeof.[nvmeof.thrasher]:done waiting before reviving 2024-09-02T12:07:57.149 INFO:tasks.nvmeof.[nvmeof.thrasher]:display and verify stats: 2024-09-02T12:07:57.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a 2024-09-02T12:07:57.180 INFO:teuthology.orchestra.run.smithi064.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service - Ceph nvmeof.nvmeof.a for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:07:57.180 INFO:teuthology.orchestra.run.smithi064.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:07:57.180 INFO:teuthology.orchestra.run.smithi064.stdout: Active: active (running) since Mon 2024-09-02 12:01:21 UTC; 6min ago 2024-09-02T12:07:57.180 INFO:teuthology.orchestra.run.smithi064.stdout: Process: 46729 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: Process: 46730 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.a/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: Main PID: 47519 (conmon) 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: Tasks: 131 (limit: 202790) 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: Memory: 1.1G 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: CPU: 6min 55.714s 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: ├─libpod-payload-5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─47521 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─47559 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─47624 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.a --gateway-address 172.21.15.64:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.64:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.a -k /etc/ceph/keyring 2024-09-02T12:07:57.181 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─48171 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout: │ └─48219 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout: └─runtime 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout: └─47519 /usr/bin/conmon --api-version 1 -c 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f -u 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f/userdata -p /run/containers/storage/overlay-containers/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: OPTIMIZED 2024-09-02T12:07:57.182 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:07:57.183 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:37516 2024-09-02T12:07:57.183 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:06:23 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [2024-09-02 12:06:23.017419] bdev_rbd.c:1039:_bdev_rbd_wait_for_latest_osdmap: *NOTICE*: rados_wait_for_latest_osdmap cluster: cluster_context_3_0 elapsed time: 0.000292 seconds 2024-09-02T12:07:57.184 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b 2024-09-02T12:07:57.217 INFO:teuthology.orchestra.run.smithi087.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service - Ceph nvmeof.nvmeof.b for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:07:57.217 INFO:teuthology.orchestra.run.smithi087.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:07:57.217 INFO:teuthology.orchestra.run.smithi087.stdout: Active: active (running) since Mon 2024-09-02 12:00:46 UTC; 7min ago 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: Process: 40870 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: Process: 40871 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.b/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: Main PID: 41182 (conmon) 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: Tasks: 130 (limit: 202788) 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: Memory: 1.1G 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: CPU: 6min 53.832s 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service 2024-09-02T12:07:57.218 INFO:teuthology.orchestra.run.smithi087.stdout: ├─libpod-payload-59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─41184 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─41187 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─41208 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.b --gateway-address 172.21.15.87:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.87:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.b -k /etc/ceph/keyring 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─42185 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: │ └─42187 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: └─runtime 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: └─41182 /usr/bin/conmon --api-version 1 -c 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 -u 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678/userdata -p /run/containers/storage/overlay-containers/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:07:57.219 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:37.138836] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:38.887886] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:07:57.220 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:42] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:59450 2024-09-02T12:07:57.221 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout:○ ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service - Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Active: inactive (dead) since Mon 2024-09-02 12:06:10 UTC; 1min 47s ago 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Duration: 5min 1.672s 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 40488 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 40489 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.c/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 46857 ExecStop=/bin/bash -c bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.c/unit.stop (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 46911 ExecStopPost=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.c/unit.poststop (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.258 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 46938 ExecStopPost=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: Main PID: 40889 (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: CPU: 5min 5.087s 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO state.py:545 (2): Cleanup OMAP on exit (gateway-client.nvmeof.nvmeof.c) 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO server.py:137 (2): Exiting the gateway process. 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[40889]: [02-Sep-2024 12:06:09] INFO utils.py:392 (2): Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvmeof.c/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvmeof.c/nvmeof-log.gz 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:09 smithi094 podman[46884]: 2024-09-02 12:06:09.312511765 +0000 UTC m=+0.229124695 container died 6c79565901db2f837e6b7d2aca3566abe4381be45c45a99d92c5109b7f167bb3 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.buildah.version=1.29.0, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.repo=https://github.com/ceph/spdk.git, url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.ceph-release=18.2.4, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, maintainer=Ceph Developers , io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.url=https://github.com/ceph/ceph-nvmeof, build-date=2024-07-30 15:47:38 UTC, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.spdk.build.target-arch=, summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, org.centos.version=9.0-21.el9, vendor=Red Hat, Inc., version=1.2.17, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.openshift.wants=ceph,rbd, release=, vcs-type=git, io.ceph.version=1.2.17, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.version=24.01, io.spdk.build.pkgdep-args=--rbd, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, architecture=x86_64, name=ceph-nvmeof, io.ceph.maintainer=Ceph Developers , io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.component=ceph-nvmeof, io.ceph.git.branch=tags/1.2.17, io.ceph.nvmeof=None, io.spdk.build.makeflags=, com.redhat.component=ubi9-container) 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:09 smithi094 podman[46884]: 2024-09-02 12:06:09.81134219 +0000 UTC m=+0.727955107 container cleanup 6c79565901db2f837e6b7d2aca3566abe4381be45c45a99d92c5109b7f167bb3 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.target-arch=, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.ceph.git.branch=tags/1.2.17, io.spdk.build.makeflags=, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., url=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.version=1.2.17, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.buildah.version=1.29.0, release=, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.summary=Ceph NVMe over Fabrics Gateway, summary=Ceph NVMe over Fabrics Gateway, io.ceph.component=ceph-nvmeof, org.centos.version=9.0-21.el9, name=ceph-nvmeof, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.openshift.wants=ceph,rbd, io.spdk.build.pkgdep-args=--rbd, io.ceph.nvmeof=None, version=1.2.17, io.spdk.build.ceph-release=18.2.4, maintainer=Ceph Developers , io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, build-date=2024-07-30 15:47:38 UTC, io.spdk.git.branch=undefined, com.redhat.component=ubi9-container, io.spdk.version=24.01, vendor=Red Hat, Inc., io.ceph.maintainer=Ceph Developers , distribution-scope=public, architecture=x86_64, io.k8s.display-name=Ceph NVMe over Fabrics Gateway) 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:09 smithi094 bash[46884]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:10 smithi094 podman[46896]: 2024-09-02 12:06:10.02130129 +0000 UTC m=+0.710585685 container remove 6c79565901db2f837e6b7d2aca3566abe4381be45c45a99d92c5109b7f167bb3 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, vendor=Red Hat, Inc., description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, url=https://github.com/ceph/ceph-nvmeof, io.spdk.version=24.01, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.buildah.version=1.29.0, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.summary=Ceph NVMe over Fabrics Gateway, version=1.2.17, io.ceph.component=ceph-nvmeof, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.openshift.wants=ceph,rbd, io.spdk.build.pkgdep-args=--rbd, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, name=ceph-nvmeof, architecture=x86_64, io.ceph.url=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, distribution-scope=public, com.redhat.component=ubi9-container, release=, io.ceph.git.branch=tags/1.2.17, io.ceph.version=1.2.17, io.spdk.build.makeflags=, io.spdk.build.target-arch=, maintainer=Ceph Developers , org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.nvmeof=None, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2024-07-30 15:47:38 UTC, org.centos.version=9.0-21.el9, io.ceph.maintainer=Ceph Developers , io.spdk.build.ceph-release=18.2.4) 2024-09-02T12:07:57.259 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:10 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Deactivated successfully. 2024-09-02T12:07:57.260 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:10 smithi094 systemd[1]: Stopped Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:07:57.260 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:06:10 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Consumed 5min 5.087s CPU time. 2024-09-02T12:07:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:07:57.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d 2024-09-02T12:07:57.294 INFO:teuthology.orchestra.run.smithi099.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service - Ceph nvmeof.nvmeof.d for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:07:57.294 INFO:teuthology.orchestra.run.smithi099.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:07:57.294 INFO:teuthology.orchestra.run.smithi099.stdout: Active: active (running) since Mon 2024-09-02 12:00:57 UTC; 7min ago 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: Process: 22398 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: Process: 22399 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.d/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: Main PID: 22636 (conmon) 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: Tasks: 130 (limit: 202788) 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: Memory: 1.1G 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: CPU: 6min 51.802s 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: ├─libpod-payload-b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22638 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22640 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22662 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.d --gateway-address 172.21.15.99:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.99:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.d -k /etc/ceph/keyring 2024-09-02T12:07:57.295 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22993 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout: │ └─23027 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout: └─runtime 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout: └─22636 /usr/bin/conmon --api-version 1 -c b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 -u b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60/userdata -p /run/containers/storage/overlay-containers/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:37.135742] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:07:57.296 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:38.885553] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:07:57.297 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:07:57.297 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:45 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:45] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:35604 2024-09-02T12:07:57.298 DEBUG:teuthology.orchestra.run.smithi038:> ceph orch ls && ceph orch ps --daemon-type nvmeof && ceph health detail && ceph -s && sudo nvme list-subsys /dev/nvme6n1 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n1 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n1 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n2 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n2 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n2 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n3 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n3 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n3 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n4 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n4 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n4 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n5 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n5 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n5 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n6 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n6 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n6 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n7 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n7 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n7 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n8 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n8 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n8 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n9 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n9 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n9 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n10 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n10 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n10 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n11 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n11 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n11 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n12 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n12 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n12 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n13 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n13 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n13 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n14 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n14 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n14 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n15 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n15 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n15 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n16 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n16 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n16 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n17 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n17 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n17 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n18 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n18 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n18 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n19 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n19 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n19 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n20 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n20 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n20 | grep 'live optimized' 2024-09-02T12:07:57.612 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-02T12:07:57.612 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 1/1 6m ago 11m smithi064=x;count:1 2024-09-02T12:07:57.613 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 6m ago 12m smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c;count:3 2024-09-02T12:07:57.613 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.mypool.mygroup0 ?:4420,5500,8009 3/4 6m ago 7m smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d 2024-09-02T12:07:57.613 INFO:teuthology.orchestra.run.smithi038.stdout:osd 8 6m ago - 2024-09-02T12:07:57.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:07:57.911 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-02T12:07:57.911 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.a smithi064 *:5500,4420,8009 running (6m) 6m ago 6m 50.1M - 1.2.17 32a3d75b7c14 5cc04673a284 2024-09-02T12:07:57.911 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.b smithi087 *:5500,4420,8009 running (7m) 6m ago 7m 47.3M - 1.2.17 32a3d75b7c14 59b7589e0a00 2024-09-02T12:07:57.911 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.c smithi094 *:5500,4420,8009 stopped 106s ago 6m - - 2024-09-02T12:07:57.911 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.d smithi099 *:5500,4420,8009 running (7m) 6m ago 7m 45.9M - 1.2.17 32a3d75b7c14 b0e3f86e16c7 2024-09-02T12:07:57.920 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:57 smithi064 ceph-mon[17163]: pgmap v430: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:57.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:07:58.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:57 smithi087 ceph-mon[21100]: pgmap v430: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:58.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:07:58.035 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:07:58.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:57 smithi094 ceph-mon[20404]: pgmap v430: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:07:58.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:07:58.177 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:07:58.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:07:58.269 INFO:teuthology.orchestra.run.smithi038.stdout:HEALTH_OK 2024-09-02T12:07:58.289 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:07:58.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:07:58.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:07:58.424 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:07:58.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:07:58.624 INFO:teuthology.orchestra.run.smithi038.stdout: cluster: 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: id: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: health: HEALTH_OK 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: services: 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: mgr: x(active, since 12m) 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 8m) 2024-09-02T12:07:58.625 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: data: 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: pools: 2 pools, 33 pgs 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: objects: 186 objects, 609 KiB 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: pgs: 33 active+clean 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: io: 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: client: 36 KiB/s rd, 2 op/s rd, 0 op/s wr 2024-09-02T12:07:58.626 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:07:58.658 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.678 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.698 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.717 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.735 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:58 smithi087 ceph-mon[21100]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:07:58.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:58 smithi087 ceph-mon[21100]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "daemon_type": "nvmeof", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:07:58.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:58 smithi087 ceph-mon[21100]: from='client.? 172.21.15.38:0/888674141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-02T12:07:58.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:58 smithi087 ceph-mon[21100]: from='client.? 172.21.15.38:0/2046968423' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T12:07:58.754 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.773 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.792 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.810 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.830 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:58 smithi094 ceph-mon[20404]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:07:58.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:58 smithi094 ceph-mon[20404]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "daemon_type": "nvmeof", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:07:58.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:58 smithi094 ceph-mon[20404]: from='client.? 172.21.15.38:0/888674141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-02T12:07:58.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:58 smithi094 ceph-mon[20404]: from='client.? 172.21.15.38:0/2046968423' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T12:07:58.848 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.867 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.885 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.904 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.923 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.941 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.960 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.979 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:58.998 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.016 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.035 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.054 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.072 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.091 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:58 smithi064 ceph-mon[17163]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:07:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:58 smithi064 ceph-mon[17163]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch ps", "daemon_type": "nvmeof", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:07:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:58 smithi064 ceph-mon[17163]: from='client.? 172.21.15.38:0/888674141' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-02T12:07:59.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:58 smithi064 ceph-mon[17163]: from='client.? 172.21.15.38:0/2046968423' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T12:07:59.110 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.129 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.147 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.166 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.185 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.204 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.222 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.241 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.260 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.278 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.297 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.316 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.334 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.372 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.390 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.409 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.428 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.447 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.466 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.484 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.503 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.521 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.539 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.558 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.577 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.595 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.614 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.633 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.744 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.763 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:07:59.764 INFO:tasks.nvmeof.[nvmeof.thrasher]:reviving nvmeof.c 2024-09-02T12:07:59.764 DEBUG:teuthology.orchestra.run.smithi094:> ceph orch daemon restart nvmeof.nvmeof.c 2024-09-02T12:08:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:07:59 smithi087 ceph-mon[21100]: pgmap v431: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:08:00.069 INFO:teuthology.orchestra.run.smithi094.stdout:Scheduled to restart nvmeof.nvmeof.c on host 'smithi094' 2024-09-02T12:08:00.081 INFO:tasks.nvmeof.[nvmeof.thrasher]:waiting for 72 secs before thrashing 2024-09-02T12:08:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:07:59 smithi064 ceph-mon[17163]: pgmap v431: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:08:00.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:07:59 smithi094 ceph-mon[20404]: pgmap v431: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1 op/s 2024-09-02T12:08:00.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:00.610 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:08:00.610 INFO:tasks.workunit.client.3.smithi094.stdout: 9.40 0.0k 32.8k 0.0k 0.0k 328.0k 0.0k nvme0n1 2024-09-02T12:08:00.610 INFO:tasks.workunit.client.3.smithi094.stdout: 4.60 0.0k 1.0M 0.0k 0.0k 10.2M 0.0k sda 2024-09-02T12:08:00.610 INFO:tasks.workunit.client.3.smithi094.stdout: 4.60 0.0k 1.0M 0.0k 0.0k 10.2M 0.0k sda1 2024-09-02T12:08:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:08:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:08:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:08:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:08:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 9.40 0.0k 32.8k 0.0k 0.0k 328.0k 0.0k dm-4 2024-09-02T12:08:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:00.614 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-09-02T12:08:01.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:00 smithi094 systemd[1]: Starting Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:08:01.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 podman[49523]: 2024-09-02 12:08:00.94937683 +0000 UTC m=+0.016189810 image pull 32a3d75b7c146d6c37b04ee3c9ba883ab88a8f7ae8f286de268d0f41ebd86a51 quay.io/ceph/nvmeof:1.2 2024-09-02T12:08:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "restart", "name": "nvmeof.nvmeof.c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:08:01.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: Schedule restart daemon nvmeof.nvmeof.c 2024-09-02T12:08:01.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:08:01.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:08:01.098 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:08:01.099 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "restart", "name": "nvmeof.nvmeof.c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:08:01.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: Schedule restart daemon nvmeof.nvmeof.c 2024-09-02T12:08:01.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:08:01.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:08:01.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:08:01.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:01 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.377 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 podman[49523]: 2024-09-02 12:08:01.126028711 +0000 UTC m=+0.192841684 container create 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.openshift.wants=ceph,rbd, io.ceph.component=ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, com.redhat.component=ubi9-container, io.spdk.build.makeflags=, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, summary=Ceph NVMe over Fabrics Gateway, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vendor=Red Hat, Inc., name=ceph-nvmeof, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.pkgdep-args=--rbd, io.spdk.version=24.01, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.buildah.version=1.29.0, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, build-date=2024-07-30 15:47:38 UTC, url=https://github.com/ceph/ceph-nvmeof, maintainer=Ceph Developers , io.spdk.build.target-arch=, io.spdk.git.branch=undefined, distribution-scope=public, io.ceph.nvmeof=None, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, org.centos.version=9.0-21.el9, io.spdk.git.repo=https://github.com/ceph/spdk.git, version=1.2.17, io.ceph.version=1.2.17, io.ceph.maintainer=Ceph Developers , org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.build.ceph-release=18.2.4, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, vcs-type=git, release=, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.branch=tags/1.2.17, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., architecture=x86_64) 2024-09-02T12:08:01.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch daemon", "action": "restart", "name": "nvmeof.nvmeof.c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:08:01.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: Schedule restart daemon nvmeof.nvmeof.c 2024-09-02T12:08:01.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:08:01.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:08:01.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:08:01.501 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:01 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:01.630 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 podman[49523]: 2024-09-02 12:08:01.376960209 +0000 UTC m=+0.443773190 container init 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, com.redhat.component=ubi9-container, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, architecture=x86_64, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, distribution-scope=public, org.centos.version=9.0-21.el9, io.spdk.build.target-arch=, io.ceph.nvmeof=None, maintainer=Ceph Developers , io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.spdk.version=24.01, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.url=https://github.com/ceph/ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, release=, io.spdk.git.branch=undefined, io.spdk.build.ceph-release=18.2.4, url=https://github.com/ceph/ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.buildah.version=1.29.0, name=ceph-nvmeof, build-date=2024-07-30 15:47:38 UTC, io.ceph.version=1.2.17, io.openshift.wants=ceph,rbd, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, vendor=Red Hat, Inc., io.ceph.git.branch=tags/1.2.17, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, summary=Ceph NVMe over Fabrics Gateway, io.ceph.maintainer=Ceph Developers , io.spdk.build.pkgdep-args=--rbd, io.ceph.component=ceph-nvmeof, io.spdk.build.makeflags=, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.git.repo=https://github.com/ceph/spdk.git, version=1.2.17, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF).) 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 podman[49523]: 2024-09-02 12:08:01.380044885 +0000 UTC m=+0.446857852 container start 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.spdk.build.target-arch=, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, summary=Ceph NVMe over Fabrics Gateway, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.maintainer=Ceph Developers , description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2024-07-30 15:47:38 UTC, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., architecture=x86_64, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.spdk.build.ceph-release=18.2.4, com.redhat.component=ubi9-container, io.spdk.git.branch=undefined, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.buildah.version=1.29.0, name=ceph-nvmeof, version=1.2.17, io.ceph.version=1.2.17, vcs-type=git, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, vendor=Red Hat, Inc., io.ceph.git.branch=tags/1.2.17, distribution-scope=public, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.version=24.01, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.build.pkgdep-args=--rbd, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.nvmeof=None, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.makeflags=, release=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.wants=ceph,rbd, url=https://github.com/ceph/ceph-nvmeof, io.ceph.component=ceph-nvmeof, maintainer=Ceph Developers , org.centos.version=9.0-21.el9) 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 bash[49523]: 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 systemd[1]: Started Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO utils.py:258 (2): Initialize gateway log level to "INFO" 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO utils.py:271 (2): Log files will be saved in /var/log/ceph/nvmeof-client.nvmeof.nvmeof.c, using rotation 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:78 (2): Using NVMeoF gateway version 1.2.17 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:81 (2): Configured SPDK version 24.01 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:84 (2): Using vstart cluster version based on 18.2.4 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:87 (2): NVMeoF gateway built on: 2024-07-30 15:47:38 UTC 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:90 (2): NVMeoF gateway Git repository: https://github.com/ceph/ceph-nvmeof 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:93 (2): NVMeoF gateway Git branch: tags/1.2.17 2024-09-02T12:08:01.631 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:96 (2): NVMeoF gateway Git commit: 887c7841f275a0cbc00eddb8a038cde3935b95ba 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:102 (2): SPDK Git repository: https://github.com/ceph/spdk.git 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:105 (2): SPDK Git branch: undefined 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:108 (2): SPDK Git commit: a16bb032516da05ea2b7c38fd0ad18e8a7190440 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:59 (2): Using configuration file /src/ceph-nvmeof.conf 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:61 (2): ====================================== Configuration file content ====================================== 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): # This file is generated by cephadm. 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [gateway] 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): name = client.nvmeof.nvmeof.c 2024-09-02T12:08:01.632 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): group = mygroup0 2024-09-02T12:08:01.888 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): addr = 172.21.15.94 2024-09-02T12:08:01.888 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): port = 5500 2024-09-02T12:08:01.888 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): enable_auth = False 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): state_update_notify = True 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): state_update_interval_sec = 5 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): enable_spdk_discovery_controller = False 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): enable_prometheus_exporter = True 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): prometheus_exporter_ssl = False 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): prometheus_port = 10008 2024-09-02T12:08:01.889 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): verify_nqns = True 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): omap_file_lock_duration = 20 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): omap_file_lock_retries = 30 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): omap_file_lock_retry_sleep_interval = 1.0 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): omap_file_update_reloads = 10 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): allowed_consecutive_spdk_ping_failures = 1 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): spdk_ping_interval_in_seconds = 2.0 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): enable_monitor_client = True 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): 2024-09-02T12:08:01.890 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [gateway-logs] 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): log_level = INFO 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): log_files_enabled = True 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): log_files_rotation_enabled = True 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): verbose_log_messages = True 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): max_log_file_size_in_mb = 10 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): max_log_files_count = 20 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): max_log_directory_backups = 10 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): log_directory = /var/log/ceph/ 2024-09-02T12:08:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [discovery] 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): addr = 172.21.15.94 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): port = 8009 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [ceph] 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): pool = mypool 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): config_file = /etc/ceph/ceph.conf 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): id = nvmeof.nvmeof.c 2024-09-02T12:08:01.892 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [mtls] 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): server_key = /server.key 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): client_key = /client.key 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): server_cert = /server.cert 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): client_cert = /client.cert 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): root_ca_cert = /root.ca.cert 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): 2024-09-02T12:08:01.893 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [spdk] 2024-09-02T12:08:01.894 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): tgt_path = /usr/local/bin/nvmf_tgt 2024-09-02T12:08:01.894 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): rpc_socket = /var/tmp/spdk.sock 2024-09-02T12:08:01.894 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): timeout = 60.0 2024-09-02T12:08:01.894 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): bdevs_per_cluster = 32 2024-09-02T12:08:01.894 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): log_level= 2024-09-02T12:08:01.894 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): protocol_log_level = WARNING 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): conn_retries = 10 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): transports = tcp 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7} 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): [monitor] 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:65 (2): timeout = 1.0 2024-09-02T12:08:01.895 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO config.py:66 (2): ======================================================================================================== 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:91 (2): Starting gateway client.nvmeof.nvmeof.c 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:162 (2): Starting serve, monitor client version: ceph version 19.0.0-4996-g0ec90b1e (0ec90b1e61a7489b13d6d8432156a0417f35db7f) squid (dev) 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO state.py:387 (2): nvmeof.mygroup0.state OMAP object already exists. 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:252 (2): Starting /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.c --gateway-address 172.21.15.94:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.94:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.c -k /etc/ceph/keyring 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:256 (2): monitor client process id: 19 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:151 (2): MonitorGroup server is listening on 172.21.15.94:5499 for group id 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:141 (2): Gateway client.nvmeof.nvmeof.c group id=2 2024-09-02T12:08:01.896 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:154 (2): Stopping the MonitorGroup server... 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 2024-09-02T12:08:01.709+0000 7f148a63a640 0 ms_deliver_dispatch: unhandled message 0x55d6a1753180 nvmeofgwmap magic: 0 from mon.2 v2:172.21.15.87:3300/0 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 2024-09-02T12:08:01.709+0000 7f148a63a640 0 ms_deliver_dispatch: unhandled message 0x55d6a1743ba0 mon_map magic: 0 from mon.2 v2:172.21.15.87:3300/0 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:157 (2): The MonitorGroup gRPC server stopped... 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:374 (2): SPDK Target Path: /usr/local/bin/nvmf_tgt 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:376 (2): SPDK Socket: /var/tmp/spdk.sock 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:382 (2): Starting /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:08:01.897 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:395 (2): SPDK process id: 58 2024-09-02T12:08:01.898 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO server.py:396 (2): Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0 2024-09-02T12:08:01.898 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: INFO: Setting log level to WARNING 2024-09-02T12:08:01.898 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:01] INFO client.py:110 (2): Setting log level to WARNING 2024-09-02T12:08:01.898 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:01.730458] Starting SPDK v24.01 / DPDK 23.11.0 initialization... 2024-09-02T12:08:01.898 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:01.730526] [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --no-pci --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid58 ] 2024-09-02T12:08:01.899 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:01.845105] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 2024-09-02T12:08:02.144 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:01.895531] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 2024-09-02T12:08:02.144 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.120415] tcp.c: 659:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init *** 2024-09-02T12:08:02.144 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO server.py:430 (2): Started SPDK with version "SPDK v24.01" 2024-09-02T12:08:02.144 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO server.py:285 (2): Discovery service process id: 60 2024-09-02T12:08:02.144 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO server.py:280 (60): Starting ceph nvmeof discovery service 2024-09-02T12:08:02.145 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-mon[20404]: pgmap v432: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:08:02.145 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:02.145 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:02.145 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:08:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:02 smithi064 ceph-mon[17163]: pgmap v432: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:08:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:02.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:02 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:08:02.395 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO cephutils.py:73 (2): Connected to Ceph with version "19.3.0-4585-gb59673c4 (b59673c44bd569f9f3db37f87bced695dec5fcbf) squid (dev)" 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:86 (2): Requested huge pages count is 2048 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:105 (2): Actual huge pages count is 4096 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:188 (2): NVMeoF bdevs per cluster: 32 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO state.py:387 (60): nvmeof.mygroup0.state OMAP object already exists. 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO discovery.py:321 (60): log pages info from omap: nvmeof.mygroup0.state 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO discovery.py:328 (60): discovery addr: 172.21.15.94 port: 8009 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO server.py:198 (2): Prometheus endpoint is enabled 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode3, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO prometheus.py:98 (2): Prometheus exporter running in http mode, listening on port 10008 2024-09-02T12:08:02.396 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:2630 (2): Received request to get gateway's info 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO prometheus.py:131 (2): Stats for all bdevs will be provided 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode2, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:506 (2): Received request to create subsystem nqn.2016-06.io.spdk:cnode1, enable_ha: True, max_namespaces: 256, context: None 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f from mypool/myimage23 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.213265] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_4_0, nonce 172.21.15.94:0/139294348, rc 0 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.213997] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_4_0, nonce 172.21.15.94:0/139294348 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_4_0' nonce='172.21.15.94:0/139294348' anagrp=4 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 1 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.228629] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f rbd disk to lun 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_bcef3a86-f7ac-46c8-8c65-95ef522f586f to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:08:02.397 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 from mypool/myimage21 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.253005] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_2_0, nonce 172.21.15.94:0/1951182161, rc 0 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.253146] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_2_0, nonce 172.21.15.94:0/1951182161 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_2_0' nonce='172.21.15.94:0/1951182161' anagrp=2 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 1 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.262195] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 rbd disk to lun 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_e68b47f7-f3f5-4ba6-aff8-bae6a4b751b0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.270998] nvmf_rpc.c: 275:rpc_nvmf_get_subsystems: *WARNING*: rpc_nvmf_get_subsystems: deprecated feature listener.transport is deprecated in favor of trtype to be removed in v24.05 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 from mypool/myimage2 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.297863] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_3_0, nonce 172.21.15.94:0/2682605606, rc 0 2024-09-02T12:08:02.398 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.298341] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_3_0, nonce 172.21.15.94:0/2682605606 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_3_0' nonce='172.21.15.94:0/2682605606' anagrp=3 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 1 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.309731] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 rbd disk to lun 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_6ffe1b68-223c-46c9-a396-3a904e32f1f4 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 from mypool/myimage15 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 2 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.336945] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 rbd disk to lun 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_849c8403-b862-46f7-82e8-a19b0a97dd90 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:08:02.399 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_1ec594ff-1157-4225-98ec-c6c41b32344b from mypool/myimage16 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.400 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.380332] bdev_rbd.c:1278:rbd_register_cluster: *NOTICE*: rbd_register_cluster cluster_context_1_0, nonce 172.21.15.94:0/2631927498, rc 0 2024-09-02T12:08:02.400 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.380405] bdev_rbd.c: 968:dump_cluster_nonce: *NOTICE*: dump_cluster_nonce cluster_context_1_0, nonce 172.21.15.94:0/2631927498 2024-09-02T12:08:02.400 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:248 (2): Allocated cluster name='cluster_context_1_0' nonce='172.21.15.94:0/2631927498' anagrp=1 2024-09-02T12:08:02.400 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 1 2024-09-02T12:08:02.400 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.392657] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b rbd disk to lun 2024-09-02T12:08:02.400 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_1ec594ff-1157-4225-98ec-c6c41b32344b to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:08:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:02 smithi087 ceph-mon[21100]: pgmap v432: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2 op/s 2024-09-02T12:08:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:02 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:02 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:02.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:02 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:08:02.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:08:02.655 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 from mypool/myimage35 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 3 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.436519] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 rbd disk to lun 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_b8506057-a178-4f80-a6be-0b5fc0a07487 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_547cce68-47ad-4227-a432-c48eb15be142 from mypool/myimage45 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 2 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.479954] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_547cce68-47ad-4227-a432-c48eb15be142 rbd disk to lun 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_547cce68-47ad-4227-a432-c48eb15be142 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:08:02.656 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 from mypool/myimage56 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 2 2024-09-02T12:08:02.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.499102] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 rbd disk to lun 2024-09-02T12:08:02.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_bc1c35d3-fccd-4a25-91b7-b79512fd3ce3 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:08:02.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:08:02.657 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_f96044ec-462e-4f71-a581-9eab30b51024 from mypool/myimage3 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 4 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.518109] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_f96044ec-462e-4f71-a581-9eab30b51024 rbd disk to lun 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_f96044ec-462e-4f71-a581-9eab30b51024 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 from mypool/myimage14 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 2 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.537375] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 rbd disk to lun 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_06e2ef3a-ca82-4bcb-8675-e2f9559d0645 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d from mypool/myimage28 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.658 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 3 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.557426] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d rbd disk to lun 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_bdeff7fa-3026-43ed-a9d1-845f00917b5d to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_e304d724-9f62-459b-b789-7137b80a0618 from mypool/myimage18 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 3 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.577016] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e304d724-9f62-459b-b789-7137b80a0618 rbd disk to lun 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_e304d724-9f62-459b-b789-7137b80a0618 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_40f6a294-315b-40eb-9839-236059ddbec0 from mypool/myimage51 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 5 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.596264] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_40f6a294-315b-40eb-9839-236059ddbec0 rbd disk to lun 2024-09-02T12:08:02.659 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_40f6a294-315b-40eb-9839-236059ddbec0 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 from mypool/myimage27 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 6 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.615949] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 rbd disk to lun 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_d78cb720-fa0a-4336-9914-d8a99e821ef9 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 from mypool/myimage41 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 3 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.635145] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 rbd disk to lun 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_4bcf6c93-7460-4a42-8ec4-a048bf9f7108 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:08:02.660 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:08:02.661 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_036643fa-ace3-4623-acfe-a405d326e7c7 from mypool/myimage4 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.661 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 4 2024-09-02T12:08:02.661 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.654622] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 rbd disk to lun 2024-09-02T12:08:02.661 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_036643fa-ace3-4623-acfe-a405d326e7c7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:08:02.907 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:08:02.907 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef from mypool/myimage50 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 4 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.676182] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef rbd disk to lun 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_4e4b44e6-d095-421d-a9ba-35a28fd6faef to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 from mypool/myimage9 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 4 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.697371] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 rbd disk to lun 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_202aeb00-7d9d-44a8-8d9e-190d60b8bd14 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_ba54d78c-a118-4885-be91-cc8c4c33830f from mypool/myimage12 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 5 2024-09-02T12:08:02.908 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.717227] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f rbd disk to lun 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_ba54d78c-a118-4885-be91-cc8c4c33830f to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 from mypool/myimage8 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 6 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.736399] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 rbd disk to lun 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_2e0e2ca4-1342-437a-9dd6-8744dde7e146 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 3 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 from mypool/myimage43 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 7 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.756305] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 rbd disk to lun 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_d96e1dcc-4b70-4b49-9e6a-63c891acca04 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 3 , context: None 2024-09-02T12:08:02.909 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 from mypool/myimage32 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 7 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.776318] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 rbd disk to lun 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_e25a419f-d57e-4dfb-a465-d40da05e44c5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode1, ana group 1, context: None 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_42eb2216-ff26-4390-90a3-97b35df81f87 from mypool/myimage20 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.910 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 8 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.796437] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 rbd disk to lun 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_42eb2216-ff26-4390-90a3-97b35df81f87 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 8 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e from mypool/myimage48 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 9 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.815818] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e rbd disk to lun 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_6fb4bc29-355c-4b03-9a1a-9c91e407d92e to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 8 , context: None 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 from mypool/myimage46 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 5 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.836377] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 rbd disk to lun 2024-09-02T12:08:02.911 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_99e45781-7fdb-4b7e-8dfa-55a8cd1396b8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e4936-9e03-425b-ba93-b65323d44402 from mypool/myimage44 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 10 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.857571] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 rbd disk to lun 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_6d3e4936-9e03-425b-ba93-b65323d44402 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 16 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 from mypool/myimage36 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 11 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.877918] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 rbd disk to lun 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_63cc09fa-ece0-493e-8a42-19c8f5106cf2 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 16 , context: None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_a995fbc3-9dbc-43b9-ab21-579813796890 from mypool/myimage34 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.912 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 6 2024-09-02T12:08:02.913 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.897665] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 rbd disk to lun 2024-09-02T12:08:02.913 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_a995fbc3-9dbc-43b9-ab21-579813796890 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:08:02.913 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:08:02.913 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:02.913 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 12 2024-09-02T12:08:03.158 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-mon[20404]: from='client.? 172.21.15.94:0/2570840633' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:08:03.158 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.158 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.159 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:08:03.159 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:08:03.159 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.917068] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 1 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_6d3e8754-98d1-4474-a559-ac49989ff693 from mypool/myimage1 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 5 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.936176] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 rbd disk to lun 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_6d3e8754-98d1-4474-a559-ac49989ff693 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 1 , context: None 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc from mypool/myimage11 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 8 2024-09-02T12:08:03.160 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.955243] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc rbd disk to lun 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_6619312d-add9-4d30-bd8d-c768d6f81dbc to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 14 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_70de1546-c18d-49bb-b824-32b6b807a5ab from mypool/myimage54 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 7 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.975178] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab rbd disk to lun 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_70de1546-c18d-49bb-b824-32b6b807a5ab to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 14 , context: None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:275 (2): Received request to create bdev bdev_127a264a-a286-4c07-95dc-3c381465fe2e from mypool/myimage5 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 6 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:02.994659] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_127a264a-a286-4c07-95dc-3c381465fe2e rbd disk to lun 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:02] INFO grpc.py:770 (2): Received request to add bdev_127a264a-a286-4c07-95dc-3c381465fe2e to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:08:03.161 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 9 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.014924] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 from mypool/myimage42 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 8 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.035127] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 rbd disk to lun 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_69fc3401-6585-446a-a9cf-f15cf86bf365 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 2 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 from mypool/myimage22 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.162 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 9 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.055123] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 rbd disk to lun 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_840dc038-a66d-4c9e-9bf9-15d7e098ba19 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 2 , context: None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_1d3b615a-3ba9-48ba-845e-70403d71658f from mypool/myimage38 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 10 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.074586] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f rbd disk to lun 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_1d3b615a-3ba9-48ba-845e-70403d71658f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 4 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 from mypool/myimage24 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 13 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.097221] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 rbd disk to lun 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_52a33712-7b0e-4b92-abad-4bd68e7cb304 to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 4 , context: None 2024-09-02T12:08:03.163 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 11 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_2b28c17b-650d-4aad-aa11-b432947561d5 from mypool/myimage31 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 10 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.119031] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 rbd disk to lun 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_2b28c17b-650d-4aad-aa11-b432947561d5 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 11 , context: None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 from mypool/myimage37 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 7 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.138068] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 rbd disk to lun 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_73c62527-dd34-41cf-8df2-f51afb05cec0 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:08:03.164 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 from mypool/myimage47 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.165 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 11 2024-09-02T12:08:03.165 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.157925] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 rbd disk to lun 2024-09-02T12:08:03.165 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_58f2df23-338d-49b1-bba5-6eee6231dbf9 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:08:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:03 smithi064 ceph-mon[17163]: from='client.? 172.21.15.94:0/2570840633' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:08:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:08:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:08:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.411 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 5 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:08:03.411 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc from mypool/myimage25 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.411 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 8 2024-09-02T12:08:03.411 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.178234] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc rbd disk to lun 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_b9ca847d-4dcb-441d-a845-22a88dd680cc to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 5 , context: None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_e0592f88-f459-49b5-b269-949b89c731f7 from mypool/myimage19 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 12 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.197812] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e0592f88-f459-49b5-b269-949b89c731f7 rbd disk to lun 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_e0592f88-f459-49b5-b269-949b89c731f7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 7 to nqn.2016-06.io.spdk:cnode1, ana group 4, context: None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b from mypool/myimage7 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 13 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.216934] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b rbd disk to lun 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_55652567-0cc3-463c-b02c-3c0bbfd57e1b to nqn.2016-06.io.spdk:cnode1 with ANA group id 4 using NSID 7 , context: None 2024-09-02T12:08:03.412 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 18 to nqn.2016-06.io.spdk:cnode3, ana group 3, context: None 2024-09-02T12:08:03.413 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 from mypool/myimage58 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.413 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 11 2024-09-02T12:08:03.413 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.235909] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 rbd disk to lun 2024-09-02T12:08:03.413 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_da5a06d9-7937-4c24-9b2b-40e24f9c74d8 to nqn.2016-06.io.spdk:cnode3 with ANA group id 3 using NSID 18 , context: None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 12 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f from mypool/myimage52 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 14 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.254988] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f rbd disk to lun 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_3bb2e008-3e5e-4ee7-bbe3-498688b9775f to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 12 , context: None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_02593951-8f1d-487f-8486-78cf08ee9025 from mypool/myimage17 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 9 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.273875] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_02593951-8f1d-487f-8486-78cf08ee9025 rbd disk to lun 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_02593951-8f1d-487f-8486-78cf08ee9025 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_88e9aca1-c840-4d93-9444-7a339bb07efb from mypool/myimage29 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.414 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 10 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.293067] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb rbd disk to lun 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_88e9aca1-c840-4d93-9444-7a339bb07efb to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 9 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f from mypool/myimage49 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 11 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.313213] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f rbd disk to lun 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_0aa81276-d941-465a-90c0-bb35fb8bbf0f to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 9 , context: None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 17 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 from mypool/myimage57 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 12 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.332380] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 rbd disk to lun 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_3f319486-9f14-4aa1-a409-a33fcca2ec41 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 17 , context: None 2024-09-02T12:08:03.415 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode3, ana group 2, context: None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_b5597828-c750-4702-aa14-9709fb7d1d91 from mypool/myimage53 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 13 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.351230] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 rbd disk to lun 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_b5597828-c750-4702-aa14-9709fb7d1d91 to nqn.2016-06.io.spdk:cnode3 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f from mypool/myimage26 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 12 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.370140] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f rbd disk to lun 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_5cc2b808-2dca-4ff9-814b-967d94ee695f to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode1, ana group 2, context: None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 from mypool/myimage13 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.416 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 14 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.389116] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 rbd disk to lun 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_189c7a10-b33e-4bb9-9a52-4e1b81407374 to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 6 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 from mypool/myimage6 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 13 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.410459] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 rbd disk to lun 2024-09-02T12:08:03.417 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_c390a368-6527-4721-81b5-0395d5ac0ff7 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 6 , context: None 2024-09-02T12:08:03.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:03.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:03 smithi087 ceph-mon[21100]: from='client.? 172.21.15.94:0/2570840633' entity='client.nvmeof.nvmeof.c' cmd={"prefix":"mon versions"} : dispatch 2024-09-02T12:08:03.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:08:03.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:08:03.500 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:08:03.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:03.590 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:03.620 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:03.694 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:03.766 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:03.789 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode2, ana group 3, context: None 2024-09-02T12:08:03.789 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 from mypool/myimage30 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 14 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.431965] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 rbd disk to lun 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_bfcbf073-8eab-4726-8aa5-9616df85bee8 to nqn.2016-06.io.spdk:cnode2 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 19 to nqn.2016-06.io.spdk:cnode2, ana group 4, context: None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_755c5504-369c-43fc-9da2-ccd5a6361150 from mypool/myimage39 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 14 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.453035] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 rbd disk to lun 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_755c5504-369c-43fc-9da2-ccd5a6361150 to nqn.2016-06.io.spdk:cnode2 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode2, ana group 1, context: None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_0768e8b5-df41-497d-9343-ac44cee11a6a from mypool/myimage40 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.473196] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a rbd disk to lun 2024-09-02T12:08:03.790 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_0768e8b5-df41-497d-9343-ac44cee11a6a to nqn.2016-06.io.spdk:cnode2 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:08:03.791 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 13 to nqn.2016-06.io.spdk:cnode2, ana group 2, context: None 2024-09-02T12:08:03.791 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 from mypool/myimage33 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.791 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_2_0' number bdevs: 15 2024-09-02T12:08:03.791 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.492249] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 rbd disk to lun 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_e7199db1-0e14-47dc-93cb-6cd97484f3b4 to nqn.2016-06.io.spdk:cnode2 with ANA group id 2 using NSID 13 , context: None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 15 to nqn.2016-06.io.spdk:cnode3, ana group 4, context: None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 from mypool/myimage55 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.511133] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 rbd disk to lun 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_d019f375-0d32-4e88-9dfc-eecb097fcb29 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 15 , context: None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:922 (2): Received request to add namespace 10 to nqn.2016-06.io.spdk:cnode1, ana group 3, context: None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:275 (2): Received request to create bdev bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 from mypool/myimage10 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_3_0' number bdevs: 15 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.532025] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 rbd disk to lun 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:770 (2): Received request to add bdev_7c1f1adf-a4f7-48b5-a3ea-3233f3291978 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 10 , context: None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None 2024-09-02T12:08:03.792 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode2, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:1731 (2): Received request to allow any host access for nqn.2016-06.io.spdk:cnode3, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.64:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.87:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.99:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.87:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.94:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:03.568634] tcp.c: 951:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 172.21.15.94 port 4420 *** 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.94:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.64:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi064 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.64:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode3 at 172.21.15.99:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi094 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.94:4420, context: None 2024-09-02T12:08:03.793 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi087 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode2 at 172.21.15.87:4420, context: None 2024-09-02T12:08:03.794 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:03] INFO grpc.py:2087 (2): Received request to create smithi099 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 172.21.15.99:4420, context: None 2024-09-02T12:08:03.801 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:03.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:03.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:03.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:04.008 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:04.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:04.250 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:04 smithi087 ceph-mon[21100]: pgmap v433: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2 op/s 2024-09-02T12:08:04.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-mon[17163]: pgmap v433: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2 op/s 2024-09-02T12:08:04.348 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:04 smithi094 ceph-mon[20404]: pgmap v433: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 2 op/s 2024-09-02T12:08:04.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:08:04] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:08:04.844 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: INACCESSIBLE 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: INACCESSIBLE 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: INACCESSIBLE 2024-09-02T12:08:04.845 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:08:04.846 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:56200 2024-09-02T12:08:05.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:05 smithi064 ceph-mon[17163]: pgmap v434: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 853 B/s wr, 28 op/s 2024-09-02T12:08:05.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:05 smithi094 ceph-mon[20404]: pgmap v434: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 853 B/s wr, 28 op/s 2024-09-02T12:08:06.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:05 smithi087 ceph-mon[21100]: pgmap v434: 33 pgs: 33 active+clean; 609 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 853 B/s wr, 28 op/s 2024-09-02T12:08:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:07 smithi087 ceph-mon[21100]: pgmap v435: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 2.5 KiB/s wr, 83 op/s 2024-09-02T12:08:08.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:07 smithi064 ceph-mon[17163]: pgmap v435: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 2.5 KiB/s wr, 83 op/s 2024-09-02T12:08:08.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-mon[20404]: pgmap v435: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 2.5 KiB/s wr, 83 op/s 2024-09-02T12:08:08.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:08:07] INFO grpc.py:820 (2): Received request to set ana states [nqn: "nqn.2016-06.io.spdk:cnode1" 2024-09-02T12:08:08.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: states { 2024-09-02T12:08:08.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: grp_id: 3 2024-09-02T12:08:08.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: state: OPTIMIZED 2024-09-02T12:08:08.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: } 2024-09-02T12:08:08.096 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: , nqn: "nqn.2016-06.io.spdk:cnode2" 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: states { 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: grp_id: 3 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: state: OPTIMIZED 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: } 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: states { 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: grp_id: 3 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: state: OPTIMIZED 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: } 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: ], , client address: IPv4 172.21.15.94:50784 2024-09-02T12:08:08.097 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:07.641069] bdev_rbd.c:1039:_bdev_rbd_wait_for_latest_osdmap: *NOTICE*: rados_wait_for_latest_osdmap cluster: cluster_context_3_0 elapsed time: 0.000331 seconds 2024-09-02T12:08:09.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:09.100 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:09.170 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:09.202 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:09.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:09.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:09.378 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:09.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:09.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:09.550 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:09.582 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:09.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:09.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:09 smithi064 ceph-mon[17163]: pgmap v436: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 5.0 KiB/s wr, 152 op/s 2024-09-02T12:08:10.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:09 smithi087 ceph-mon[21100]: pgmap v436: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 5.0 KiB/s wr, 152 op/s 2024-09-02T12:08:10.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:09 smithi094 ceph-mon[20404]: pgmap v436: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 5.0 KiB/s wr, 152 op/s 2024-09-02T12:08:10.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:10.610 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 46.40 0.0k 212.2k 0.0k 0.0k 2.1M 0.0k nvme0n1 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 24.10 0.0k 1.6M 0.0k 0.0k 15.7M 0.0k sda 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 24.10 0.0k 1.6M 0.0k 0.0k 15.7M 0.0k sda1 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 15.60 0.0k 62.4k 0.0k 0.0k 624.0k 0.0k dm-1 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 10.00 0.0k 40.0k 0.0k 0.0k 400.0k 0.0k dm-2 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 13.20 0.0k 52.8k 0.0k 0.0k 528.0k 0.0k dm-3 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 13.60 0.0k 57.0k 0.0k 0.0k 570.0k 0.0k dm-4 2024-09-02T12:08:10.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:11 smithi087 ceph-mon[21100]: pgmap v437: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 5.0 KiB/s wr, 153 op/s 2024-09-02T12:08:12.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:11 smithi064 ceph-mon[17163]: pgmap v437: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 5.0 KiB/s wr, 153 op/s 2024-09-02T12:08:12.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:11 smithi094 ceph-mon[20404]: pgmap v437: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 5.0 KiB/s wr, 153 op/s 2024-09-02T12:08:14.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:13 smithi087 ceph-mon[21100]: pgmap v438: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 6.1 KiB/s wr, 154 op/s 2024-09-02T12:08:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:13 smithi064 ceph-mon[17163]: pgmap v438: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 6.1 KiB/s wr, 154 op/s 2024-09-02T12:08:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:13 smithi094 ceph-mon[20404]: pgmap v438: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 6.1 KiB/s wr, 154 op/s 2024-09-02T12:08:14.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:14.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:14.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:14.778 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:14.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:14.925 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:14.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:15.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:15.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:15.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:15.162 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:15.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:16.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:15 smithi087 ceph-mon[21100]: pgmap v439: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 7.5 KiB/s wr, 157 op/s 2024-09-02T12:08:16.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:15 smithi064 ceph-mon[17163]: pgmap v439: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 7.5 KiB/s wr, 157 op/s 2024-09-02T12:08:16.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:15 smithi094 ceph-mon[20404]: pgmap v439: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 7.5 KiB/s wr, 157 op/s 2024-09-02T12:08:18.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:17 smithi087 ceph-mon[21100]: pgmap v440: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 7.9 KiB/s wr, 135 op/s 2024-09-02T12:08:18.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:17 smithi064 ceph-mon[17163]: pgmap v440: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 7.9 KiB/s wr, 135 op/s 2024-09-02T12:08:18.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:17 smithi094 ceph-mon[20404]: pgmap v440: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 145 KiB/s rd, 7.9 KiB/s wr, 135 op/s 2024-09-02T12:08:20.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:19 smithi087 ceph-mon[21100]: pgmap v441: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 6.2 KiB/s wr, 79 op/s 2024-09-02T12:08:20.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:19 smithi064 ceph-mon[17163]: pgmap v441: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 6.2 KiB/s wr, 79 op/s 2024-09-02T12:08:20.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:19 smithi094 ceph-mon[20404]: pgmap v441: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 6.2 KiB/s wr, 79 op/s 2024-09-02T12:08:20.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:20.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:20.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:20.358 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:20.430 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:20.503 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:20.535 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:20.606 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 19.30 0.0k 82.0k 0.0k 0.0k 820.5k 0.0k nvme0n1 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 10.30 0.0k 708.8k 0.0k 0.0k 6.9M 0.0k sda 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 10.30 0.0k 708.8k 0.0k 0.0k 6.9M 0.0k sda1 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 5.40 0.0k 21.6k 0.0k 0.0k 216.0k 0.0k dm-1 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 3.70 0.0k 14.8k 0.0k 0.0k 148.0k 0.0k dm-2 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 4.70 0.0k 18.8k 0.0k 0.0k 188.0k 0.0k dm-3 2024-09-02T12:08:20.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.80 0.0k 26.9k 0.0k 0.0k 268.5k 0.0k dm-4 2024-09-02T12:08:20.612 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:20.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:20.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:20.703 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:20.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:22.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:21 smithi087 ceph-mon[21100]: pgmap v442: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 11 op/s 2024-09-02T12:08:22.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:21 smithi064 ceph-mon[17163]: pgmap v442: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 11 op/s 2024-09-02T12:08:22.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:21 smithi094 ceph-mon[20404]: pgmap v442: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3.7 KiB/s wr, 11 op/s 2024-09-02T12:08:24.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:23 smithi087 ceph-mon[21100]: pgmap v443: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-09-02T12:08:24.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:23 smithi064 ceph-mon[17163]: pgmap v443: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-09-02T12:08:24.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:23 smithi094 ceph-mon[20404]: pgmap v443: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 3.7 KiB/s wr, 10 op/s 2024-09-02T12:08:25.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:25.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:25.866 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:25.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:25.960 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:25 smithi087 ceph-mon[21100]: pgmap v444: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-02T12:08:25.971 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:26.043 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:26.065 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:25 smithi094 ceph-mon[20404]: pgmap v444: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-02T12:08:26.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:26.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:25 smithi064 ceph-mon[17163]: pgmap v444: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-02T12:08:26.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:26.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:26.248 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:26.279 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:26.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:28.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:27 smithi087 ceph-mon[21100]: pgmap v445: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-02T12:08:28.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:27 smithi064 ceph-mon[17163]: pgmap v445: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-02T12:08:28.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:27 smithi094 ceph-mon[20404]: pgmap v445: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-09-02T12:08:30.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:29 smithi087 ceph-mon[21100]: pgmap v446: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:30.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:29 smithi064 ceph-mon[17163]: pgmap v446: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:30.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:29 smithi094 ceph-mon[20404]: pgmap v446: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:30.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.20 0.0k 22.9k 0.0k 0.0k 229.0k 0.0k nvme0n1 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 4.40 0.0k 1.5M 0.0k 0.0k 14.8M 0.0k sda 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 4.40 0.0k 1.5M 0.0k 0.0k 14.8M 0.0k sda1 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.20 0.0k 22.9k 0.0k 0.0k 229.0k 0.0k dm-4 2024-09-02T12:08:30.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:31.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:31.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:31.441 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:31.473 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:31.545 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:31.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:31.649 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:31.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:31.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:31 smithi087 ceph-mon[21100]: pgmap v447: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:31.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:31.821 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:31.853 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:31.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:31.904 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:31 smithi064 ceph-mon[17163]: pgmap v447: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:32.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:31 smithi094 ceph-mon[20404]: pgmap v447: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:34.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:33 smithi087 ceph-mon[21100]: pgmap v448: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:34.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:33 smithi064 ceph-mon[17163]: pgmap v448: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:34.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:33 smithi094 ceph-mon[20404]: pgmap v448: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:36.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:35 smithi087 ceph-mon[21100]: pgmap v449: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:36.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:35 smithi064 ceph-mon[17163]: pgmap v449: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:36.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:35 smithi094 ceph-mon[20404]: pgmap v449: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:36.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:36.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:37.016 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:37.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:37.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:37.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:37.226 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:37.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:37.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:37.398 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:37.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:37.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:37.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:37 smithi087 ceph-mon[21100]: pgmap v450: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:38.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:37 smithi064 ceph-mon[17163]: pgmap v450: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:38.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:37 smithi094 ceph-mon[20404]: pgmap v450: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:40.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:39 smithi087 ceph-mon[21100]: pgmap v451: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:40.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:39 smithi064 ceph-mon[17163]: pgmap v451: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:40.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:39 smithi094 ceph-mon[20404]: pgmap v451: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.90 0.0k 22.8k 0.0k 0.0k 228.0k 0.0k nvme0n1 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 5.10 0.0k 1.3M 0.0k 0.0k 13.5M 0.0k sda 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 5.10 0.0k 1.3M 0.0k 0.0k 13.5M 0.0k sda1 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.90 0.0k 22.8k 0.0k 0.0k 228.0k 0.0k dm-4 2024-09-02T12:08:40.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:42.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:41 smithi087 ceph-mon[21100]: pgmap v452: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:42.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:41 smithi064 ceph-mon[17163]: pgmap v452: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:42.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:41 smithi094 ceph-mon[20404]: pgmap v452: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:42.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:42.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:42.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:42.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:42.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:42.771 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:42.802 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:42.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:42.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:42.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:43.007 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:43.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:44.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:43 smithi087 ceph-mon[21100]: pgmap v453: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:44.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:43 smithi064 ceph-mon[17163]: pgmap v453: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:44.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:43 smithi094 ceph-mon[20404]: pgmap v453: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:46.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:45 smithi087 ceph-mon[21100]: pgmap v454: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:46.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:45 smithi064 ceph-mon[17163]: pgmap v454: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:46.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:45 smithi094 ceph-mon[20404]: pgmap v454: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:48.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:47 smithi087 ceph-mon[21100]: pgmap v455: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:48.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:48.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:47 smithi064 ceph-mon[17163]: pgmap v455: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:48.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:47 smithi094 ceph-mon[20404]: pgmap v455: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:48.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:48.171 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:48.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:48.275 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:48.348 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:48.379 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:48.450 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:48.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:48.551 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:48.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:48.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:50.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:49 smithi087 ceph-mon[21100]: pgmap v456: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:50.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:49 smithi064 ceph-mon[17163]: pgmap v456: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:50.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:49 smithi094 ceph-mon[20404]: pgmap v456: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:50.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:50.610 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.50 0.0k 25.8k 0.0k 0.0k 257.5k 0.0k nvme0n1 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 4.60 0.0k 664.0k 0.0k 0.0k 6.5M 0.0k sda 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 4.60 0.0k 664.0k 0.0k 0.0k 6.5M 0.0k sda1 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 7.60 0.0k 25.8k 0.0k 0.0k 257.5k 0.0k dm-4 2024-09-02T12:08:50.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:08:52.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:51 smithi087 ceph-mon[21100]: pgmap v457: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:52.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:51 smithi064 ceph-mon[17163]: pgmap v457: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:52.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:51 smithi094 ceph-mon[20404]: pgmap v457: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:53.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:53.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:53.747 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:53.780 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:53.852 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:53.925 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:53.946 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:53 smithi094 ceph-mon[20404]: pgmap v458: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:53.956 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:54.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:53 smithi087 ceph-mon[21100]: pgmap v458: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:54.027 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:54.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:53 smithi064 ceph-mon[17163]: pgmap v458: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:54.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:54.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:54.161 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:54.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:56.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:55 smithi087 ceph-mon[21100]: pgmap v459: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:56.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:55 smithi064 ceph-mon[17163]: pgmap v459: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:56.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:55 smithi094 ceph-mon[20404]: pgmap v459: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:08:58.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:57 smithi087 ceph-mon[21100]: pgmap v460: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:58.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:57 smithi064 ceph-mon[17163]: pgmap v460: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:58.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:57 smithi094 ceph-mon[20404]: pgmap v460: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:08:59.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:08:59.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:08:59.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:08:59.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:08:59.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:08:59.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:08:59.531 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:08:59.603 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:08:59.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:08:59.705 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:08:59.737 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:08:59.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:08:59.846 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:08:59 smithi094 ceph-mon[20404]: pgmap v461: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:00.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:08:59 smithi087 ceph-mon[21100]: pgmap v461: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:00.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:08:59 smithi064 ceph-mon[17163]: pgmap v461: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:00.610 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.80 0.0k 22.4k 0.0k 0.0k 224.0k 0.0k nvme0n1 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 5.10 0.0k 1.2M 0.0k 0.0k 11.9M 0.0k sda 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 5.10 0.0k 1.2M 0.0k 0.0k 11.9M 0.0k sda1 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-0 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-1 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-2 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 0.00 0.0k 0.0k 0.0k 0.0k 0.0k 0.0k dm-3 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 6.80 0.0k 22.4k 0.0k 0.0k 224.0k 0.0k dm-4 2024-09-02T12:09:00.611 INFO:tasks.workunit.client.3.smithi094.stdout: 2024-09-02T12:09:00.616 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-09-02T12:09:00.616 INFO:tasks.workunit:Stopping ['nvmeof/fio_test.sh --rbd_iostat'] on client.3... 2024-09-02T12:09:00.616 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.3 /home/ubuntu/cephtest/clone.client.3 2024-09-02T12:09:00.906 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test nvmeof/fio_test.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 IOSTAT_INTERVAL=10 RBD_POOL=mypool RUNTIME=600 timeout 30m /home/ubuntu/cephtest/clone.client.3/qa/workunits/nvmeof/fio_test.sh --rbd_iostat' 2024-09-02T12:09:01.136 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=045a301960eb41d593a650e510901aa0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_vallariag_ceph_6bdd3521695a0621868ce5034eaba1fd0b6e3cd1/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test nvmeof/fio_test.sh) on smithi094 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 IOSTAT_INTERVAL=10 RBD_POOL=mypool RUNTIME=600 timeout 30m /home/ubuntu/cephtest/clone.client.3/qa/workunits/nvmeof/fio_test.sh --rbd_iostat' 2024-09-02T12:09:01.138 DEBUG:teuthology.run_tasks:Unwinding manager nvmeof.thrash 2024-09-02T12:09:01.146 INFO:tasks.nvmeof:joining nvmeof.thrash 2024-09-02T12:09:02.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:01 smithi087 ceph-mon[21100]: pgmap v462: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:02.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:01 smithi064 ceph-mon[17163]: pgmap v462: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:02.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:01 smithi094 ceph-mon[20404]: pgmap v462: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:04.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: pgmap v463: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:04.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:09:04.001 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:03 smithi087 ceph-mon[21100]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:09:04.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: pgmap v463: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:04.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:09:04.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:09:04.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:09:04.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:09:04.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:03 smithi064 ceph-mon[17163]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:09:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: pgmap v463: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T12:09:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T12:09:04.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:09:04.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T12:09:04.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T12:09:04.097 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:03 smithi094 ceph-mon[20404]: from='mgr.14150 172.21.15.64:0/1769968040' entity='mgr.x' 2024-09-02T12:09:04.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:04.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:04.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:04.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:04.994 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:04 smithi087 ceph-mon[21100]: Adjusting osd_memory_target on smithi087 to 5696M 2024-09-02T12:09:05.005 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:05.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:05.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:04 smithi064 ceph-mon[17163]: Adjusting osd_memory_target on smithi087 to 5696M 2024-09-02T12:09:05.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:04 smithi094 ceph-mon[20404]: Adjusting osd_memory_target on smithi087 to 5696M 2024-09-02T12:09:05.110 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:05.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:05.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:05.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:05.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:05.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:06.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:05 smithi087 ceph-mon[21100]: pgmap v464: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:06.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:05 smithi064 ceph-mon[17163]: pgmap v464: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:06.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:05 smithi094 ceph-mon[20404]: pgmap v464: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:08.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:07 smithi087 ceph-mon[21100]: pgmap v465: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:08.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:07 smithi064 ceph-mon[17163]: pgmap v465: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:08.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:07 smithi094 ceph-mon[20404]: pgmap v465: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:10.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:09 smithi087 ceph-mon[21100]: pgmap v466: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:10.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:09 smithi064 ceph-mon[17163]: pgmap v466: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:10.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:09 smithi094 ceph-mon[20404]: pgmap v466: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:10.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:10.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:10.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:10.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:10.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:10.657 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:10.689 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:10.760 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:10.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:10.861 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:10.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:10.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:12.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:11 smithi087 ceph-mon[21100]: pgmap v467: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:12.082 INFO:tasks.nvmeof.[nvmeof.thrasher]:done waiting before thrashing 2024-09-02T12:09:12.082 INFO:tasks.nvmeof.[nvmeof.thrasher]:display and verify stats: 2024-09-02T12:09:12.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a 2024-09-02T12:09:12.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:11 smithi064 ceph-mon[17163]: pgmap v467: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:12.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:11 smithi094 ceph-mon[20404]: pgmap v467: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 3 op/s 2024-09-02T12:09:12.112 INFO:teuthology.orchestra.run.smithi064.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service - Ceph nvmeof.nvmeof.a for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:09:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:09:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: Active: active (running) since Mon 2024-09-02 12:01:21 UTC; 7min ago 2024-09-02T12:09:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: Process: 46729 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.112 INFO:teuthology.orchestra.run.smithi064.stdout: Process: 46730 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.a/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: Main PID: 47519 (conmon) 2024-09-02T12:09:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: Tasks: 131 (limit: 202790) 2024-09-02T12:09:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: Memory: 1.1G 2024-09-02T12:09:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: CPU: 8min 10.899s 2024-09-02T12:09:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service 2024-09-02T12:09:12.113 INFO:teuthology.orchestra.run.smithi064.stdout: ├─libpod-payload-5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─47521 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─47559 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─47624 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.a --gateway-address 172.21.15.64:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.64:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.a -k /etc/ceph/keyring 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: │ ├─48171 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: │ └─48219 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: └─runtime 2024-09-02T12:09:12.114 INFO:teuthology.orchestra.run.smithi064.stdout: └─47519 /usr/bin/conmon --api-version 1 -c 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f -u 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f/userdata -p /run/containers/storage/overlay-containers/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: INACCESSIBLE 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:09:12.115 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: states { 2024-09-02T12:09:12.116 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: grp_id: 3 2024-09-02T12:09:12.116 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: state: INACCESSIBLE 2024-09-02T12:09:12.116 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: } 2024-09-02T12:09:12.116 INFO:teuthology.orchestra.run.smithi064.stdout:Sep 02 12:08:04 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ], , client address: IPv4 172.21.15.64:56200 2024-09-02T12:09:12.117 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b 2024-09-02T12:09:12.147 INFO:teuthology.orchestra.run.smithi087.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service - Ceph nvmeof.nvmeof.b for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:09:12.147 INFO:teuthology.orchestra.run.smithi087.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:09:12.147 INFO:teuthology.orchestra.run.smithi087.stdout: Active: active (running) since Mon 2024-09-02 12:00:46 UTC; 8min ago 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: Process: 40870 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: Process: 40871 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.b/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: Main PID: 41182 (conmon) 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: Tasks: 130 (limit: 202788) 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: Memory: 1.1G 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: CPU: 8min 9.035s 2024-09-02T12:09:12.148 INFO:teuthology.orchestra.run.smithi087.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: ├─libpod-payload-59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─41184 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─41187 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─41208 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.b --gateway-address 172.21.15.87:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.87:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.b -k /etc/ceph/keyring 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: │ ├─42185 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: │ └─42187 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: └─runtime 2024-09-02T12:09:12.149 INFO:teuthology.orchestra.run.smithi087.stdout: └─41182 /usr/bin/conmon --api-version 1 -c 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 -u 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678/userdata -p /run/containers/storage/overlay-containers/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:37.138836] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:37 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:09:12.150 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:09:12.151 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:09:12.151 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [2024-09-02 12:05:38.887886] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:09:12.151 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:09:12.151 INFO:teuthology.orchestra.run.smithi087.stdout:Sep 02 12:05:42 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:05:42] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:59450 2024-09-02T12:09:12.152 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c 2024-09-02T12:09:12.181 INFO:teuthology.orchestra.run.smithi094.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service - Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:09:12.181 INFO:teuthology.orchestra.run.smithi094.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:09:12.181 INFO:teuthology.orchestra.run.smithi094.stdout: Active: active (running) since Mon 2024-09-02 12:08:01 UTC; 1min 10s ago 2024-09-02T12:09:12.181 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 49476 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: Process: 49477 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.c/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: Main PID: 49537 (conmon) 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: Tasks: 130 (limit: 202789) 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: Memory: 142.2M 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: CPU: 1min 11.072s 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service 2024-09-02T12:09:12.182 INFO:teuthology.orchestra.run.smithi094.stdout: ├─libpod-payload-64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: │ ├─49539 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: │ ├─49541 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: │ ├─49625 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.c --gateway-address 172.21.15.94:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.94:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.c -k /etc/ceph/keyring 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: │ ├─49664 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: │ └─49763 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: └─runtime 2024-09-02T12:09:12.183 INFO:teuthology.orchestra.run.smithi094.stdout: └─49537 /usr/bin/conmon --api-version 1 -c 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 -u 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6/userdata -p /run/containers/storage/overlay-containers/64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 2024-09-02T12:09:12.184 INFO:teuthology.orchestra.run.smithi094.stdout: 2024-09-02T12:09:12.184 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: grp_id: 3 2024-09-02T12:09:12.184 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: state: OPTIMIZED 2024-09-02T12:09:12.185 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: } 2024-09-02T12:09:12.185 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: , nqn: "nqn.2016-06.io.spdk:cnode3" 2024-09-02T12:09:12.185 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: states { 2024-09-02T12:09:12.185 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: grp_id: 3 2024-09-02T12:09:12.188 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: state: OPTIMIZED 2024-09-02T12:09:12.188 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: } 2024-09-02T12:09:12.188 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: ], , client address: IPv4 172.21.15.94:50784 2024-09-02T12:09:12.188 INFO:teuthology.orchestra.run.smithi094.stdout:Sep 02 12:08:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [2024-09-02 12:08:07.641069] bdev_rbd.c:1039:_bdev_rbd_wait_for_latest_osdmap: *NOTICE*: rados_wait_for_latest_osdmap cluster: cluster_context_3_0 elapsed time: 0.000331 seconds 2024-09-02T12:09:12.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d 2024-09-02T12:09:12.220 INFO:teuthology.orchestra.run.smithi099.stdout:● ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service - Ceph nvmeof.nvmeof.d for e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:09:12.220 INFO:teuthology.orchestra.run.smithi099.stdout: Loaded: loaded (/etc/systemd/system/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@.service; enabled; preset: disabled) 2024-09-02T12:09:12.220 INFO:teuthology.orchestra.run.smithi099.stdout: Active: active (running) since Mon 2024-09-02 12:00:57 UTC; 8min ago 2024-09-02T12:09:12.220 INFO:teuthology.orchestra.run.smithi099.stdout: Process: 22398 ExecStartPre=/bin/rm -f /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service-pid /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service-cid (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.220 INFO:teuthology.orchestra.run.smithi099.stdout: Process: 22399 ExecStart=/bin/bash /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/nvmeof.nvmeof.d/unit.run (code=exited, status=0/SUCCESS) 2024-09-02T12:09:12.220 INFO:teuthology.orchestra.run.smithi099.stdout: Main PID: 22636 (conmon) 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: Tasks: 130 (limit: 202788) 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: Memory: 1.1G 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: CPU: 8min 6.954s 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: CGroup: /system.slice/system-ceph\x2de8e22764\x2d6921\x2d11ef\x2dbcd4\x2dc7b262605968.slice/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: ├─libpod-payload-b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22638 /run/podman-init -- python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.221 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22640 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22662 /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.nvmeof.d --gateway-address 172.21.15.99:5500 --gateway-pool mypool --gateway-group mygroup0 --monitor-group-address 172.21.15.99:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.nvmeof.d -k /etc/ceph/keyring 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout: │ ├─22993 /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout: │ └─23027 python3 -m control -c /src/ceph-nvmeof.conf 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout: └─runtime 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout: └─22636 /usr/bin/conmon --api-version 1 -c b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 -u b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60/userdata -p /run/containers/storage/overlay-containers/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60/userdata/pidfile -n ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d --exit-dir /run/libpod/exits --persist-dir /run/libpod/persist/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60/userdata/oci-log --conmon-pidfile /run/ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service-pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg "" --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-02T12:09:12.222 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:275 (2): Received request to create bdev bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 from mypool/myimage59 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_4_0' number bdevs: 15 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:37.135742] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 rbd disk to lun 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:37 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:37] INFO grpc.py:770 (2): Received request to add bdev_3a7171f0-c316-4798-8db4-433cac4e7b05 to nqn.2016-06.io.spdk:cnode3 with ANA group id 4 using NSID 19 , context: None 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:922 (2): Received request to add namespace 20 to nqn.2016-06.io.spdk:cnode3, ana group 1, context: None 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:275 (2): Received request to create bdev bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 from mypool/myimage60 (size 0 bytes) with block size 512, will not create image if doesn't exist, context=None 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:207 (2): get_cluster cluster_name='cluster_context_1_0' number bdevs: 15 2024-09-02T12:09:12.223 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [2024-09-02 12:05:38.885553] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 rbd disk to lun 2024-09-02T12:09:12.224 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:38 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:38] INFO grpc.py:770 (2): Received request to add bdev_22f8f0f6-9933-4a95-9582-0f374f94cad7 to nqn.2016-06.io.spdk:cnode3 with ANA group id 1 using NSID 20 , context: None 2024-09-02T12:09:12.224 INFO:teuthology.orchestra.run.smithi099.stdout:Sep 02 12:05:45 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:05:45] INFO grpc.py:2395 (2): Received request to list all subsystems, context: , client address: IPv4 172.21.15.38:35604 2024-09-02T12:09:12.225 DEBUG:teuthology.orchestra.run.smithi038:> ceph orch ls && ceph orch ps --daemon-type nvmeof && ceph health detail && ceph -s && sudo nvme list-subsys /dev/nvme6n1 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n1 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n1 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n2 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n2 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n2 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n3 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n3 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n3 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n4 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n4 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n4 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n5 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n5 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n5 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n6 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n6 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n6 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n7 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n7 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n7 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n8 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n8 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n8 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n9 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n9 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n9 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n10 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n10 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n10 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n11 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n11 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n11 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n12 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n12 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n12 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n13 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n13 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n13 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n14 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n14 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n14 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n15 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n15 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n15 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n16 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n16 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n16 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n17 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n17 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n17 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n18 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n18 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n18 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n19 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n19 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n19 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme6n20 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme2n20 | grep 'live optimized' && sudo nvme list-subsys /dev/nvme10n20 | grep 'live optimized' 2024-09-02T12:09:12.535 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-02T12:09:12.535 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 1/1 7m ago 12m smithi064=x;count:1 2024-09-02T12:09:12.536 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 7m ago 13m smithi064:172.21.15.64=a;smithi087:172.21.15.87=b;smithi094:172.21.15.94=c;count:3 2024-09-02T12:09:12.536 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.mypool.mygroup0 ?:4420,5500,8009 4/4 7m ago 8m smithi064=nvmeof.a;smithi087=nvmeof.b;smithi094=nvmeof.c;smithi099=nvmeof.d 2024-09-02T12:09:12.536 INFO:teuthology.orchestra.run.smithi038.stdout:osd 8 7m ago - 2024-09-02T12:09:12.832 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-02T12:09:12.832 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.a smithi064 *:5500,4420,8009 running (7m) 7m ago 7m 50.1M - 1.2.17 32a3d75b7c14 5cc04673a284 2024-09-02T12:09:12.832 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.b smithi087 *:5500,4420,8009 running (8m) 7m ago 8m 47.3M - 1.2.17 32a3d75b7c14 59b7589e0a00 2024-09-02T12:09:12.832 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.c smithi094 *:5500,4420,8009 running (71s) 69s ago 8m 49.8M - 1.2.17 32a3d75b7c14 64b1f23ef763 2024-09-02T12:09:12.833 INFO:teuthology.orchestra.run.smithi038.stdout:nvmeof.nvmeof.d smithi099 *:5500,4420,8009 running (8m) 7m ago 8m 45.9M - 1.2.17 32a3d75b7c14 b0e3f86e16c7 2024-09-02T12:09:13.186 INFO:teuthology.orchestra.run.smithi038.stdout:HEALTH_OK 2024-09-02T12:09:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: cluster: 2024-09-02T12:09:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: id: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:09:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: health: HEALTH_OK 2024-09-02T12:09:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:09:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: services: 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: mon: 3 daemons, quorum a,c,b (age 13m) 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: mgr: x(active, since 14m) 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: data: 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: pools: 2 pools, 33 pgs 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: objects: 186 objects, 609 KiB 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: usage: 239 MiB used, 715 GiB / 715 GiB avail 2024-09-02T12:09:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: pgs: 33 active+clean 2024-09-02T12:09:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:09:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: io: 2024-09-02T12:09:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: client: 32 KiB/s rd, 2 op/s rd, 0 op/s wr 2024-09-02T12:09:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-02T12:09:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.586 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.606 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.625 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme8 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.643 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme4 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.662 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme12 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.681 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.699 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.718 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.737 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.755 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.774 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.792 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.811 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.830 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.848 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme8 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.867 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme4 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.885 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme12 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.979 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:13.997 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:13 smithi087 ceph-mon[21100]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:09:14.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:13 smithi087 ceph-mon[21100]: pgmap v468: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:14.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:13 smithi087 ceph-mon[21100]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "daemon_type": "nvmeof", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:09:14.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:13 smithi087 ceph-mon[21100]: from='client.? 172.21.15.38:0/3020918417' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-02T12:09:14.000 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:13 smithi087 ceph-mon[21100]: from='client.? 172.21.15.38:0/715563165' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T12:09:14.016 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.034 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.053 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.072 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme8 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme4 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:13 smithi064 ceph-mon[17163]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:09:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:13 smithi064 ceph-mon[17163]: pgmap v468: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:13 smithi064 ceph-mon[17163]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "daemon_type": "nvmeof", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:09:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:13 smithi064 ceph-mon[17163]: from='client.? 172.21.15.38:0/3020918417' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-02T12:09:14.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:13 smithi064 ceph-mon[17163]: from='client.? 172.21.15.38:0/715563165' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T12:09:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:13 smithi094 ceph-mon[20404]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:09:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:13 smithi094 ceph-mon[20404]: pgmap v468: 33 pgs: 33 active+clean; 609 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2 op/s 2024-09-02T12:09:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:13 smithi094 ceph-mon[20404]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "daemon_type": "nvmeof", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T12:09:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:13 smithi094 ceph-mon[20404]: from='client.? 172.21.15.38:0/3020918417' entity='client.admin' cmd={"prefix": "health", "detail": "detail"} : dispatch 2024-09-02T12:09:14.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:13 smithi094 ceph-mon[20404]: from='client.? 172.21.15.38:0/715563165' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T12:09:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme12 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.203 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.221 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.240 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.259 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.277 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.296 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme8 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.315 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme4 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.333 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme12 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.352 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.371 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.390 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.410 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.430 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.449 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.467 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme7 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.486 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme3 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.505 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme11 tcp traddr=172.21.15.87,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.524 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme8 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.542 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme4 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.560 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme12 tcp traddr=172.21.15.94,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.579 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme9 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.598 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme5 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.617 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme13 tcp traddr=172.21.15.99,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.635 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme6 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.656 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme2 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.673 INFO:teuthology.orchestra.run.smithi038.stdout: +- nvme10 tcp traddr=172.21.15.64,trsvcid=4420,src_addr=172.21.15.38 live optimized 2024-09-02T12:09:14.674 INFO:tasks.nvmeof.[nvmeof.thrasher]:Thrasher summary: 2024-09-02T12:09:14.674 INFO:tasks.nvmeof.[nvmeof.thrasher]:nvmeof.c was thrashed 1 times 2024-09-02T12:09:14.674 INFO:tasks.nvmeof.[nvmeof.thrasher]:Iteration 0: thrashed- nvmeof.c (by ceph_daemon_stop); 2024-09-02T12:09:14.674 INFO:tasks.nvmeof:done joining 2024-09-02T12:09:14.674 DEBUG:teuthology.run_tasks:Unwinding manager nvmeof 2024-09-02T12:09:14.685 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-02T12:09:14.720 INFO:tasks.cephadm:Teardown begin 2024-09-02T12:09:14.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:09:14.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:09:14.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:09:14.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:09:14.819 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:09:14.843 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-02T12:09:14.844 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T12:09:14.858 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T12:09:14.873 DEBUG:teuthology.orchestra.run.smithi087:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T12:09:14.888 DEBUG:teuthology.orchestra.run.smithi094:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T12:09:14.903 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T12:09:14.918 INFO:tasks.cephadm:Stopping all daemons... 2024-09-02T12:09:14.918 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-02T12:09:14.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:15.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:14 smithi064 systemd[1]: Stopping Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:15.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a[17159]: 2024-09-02T12:09:15.203+0000 7f2bcc6bd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:15.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:15 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a[17159]: 2024-09-02T12:09:15.203+0000 7f2bcc6bd640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-02T12:09:15.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:15 smithi064 podman[60022]: 2024-09-02 12:09:15.336224308 +0000 UTC m=+0.234806456 container died 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-02T12:09:15.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:15.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:16.065 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:16.098 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:16.170 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:16.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:15 smithi064 podman[60022]: 2024-09-02 12:09:15.90425923 +0000 UTC m=+0.802841376 container cleanup 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:16.190 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:15 smithi064 bash[60022]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a 2024-09-02T12:09:16.243 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:16.274 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:16.346 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:16.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service' 2024-09-02T12:09:16.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:16.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:16 smithi064 podman[60035]: 2024-09-02 12:09:16.189734313 +0000 UTC m=+0.850813014 container remove 538852d87dbbc6a9c9be7289e81aaec82326de34fc5d174e80290b3faabc12c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-02T12:09:16.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:16 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-02T12:09:16.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:16 smithi064 systemd[1]: Stopped Ceph mon.a for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:09:16.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Sep 02 12:09:16 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a.service: Consumed 9.106s CPU time. 2024-09-02T12:09:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:16.450 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:16.482 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:16.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:16.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:16.523 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-02T12:09:16.524 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-02T12:09:16.524 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:16.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-02T12:09:16.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:16 smithi087 systemd[1]: Stopping Ceph mon.b for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:17.140 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-b[21096]: 2024-09-02T12:09:16.885+0000 7f4cf40bf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:17.141 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:16 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-b[21096]: 2024-09-02T12:09:16.885+0000 7f4cf40bf640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-02T12:09:17.456 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:17 smithi087 podman[50450]: 2024-09-02 12:09:17.140148189 +0000 UTC m=+0.412562908 container died e7448cffa948343e4a301e1f20b654f0c8a803e16431427a704f7a1acaefa35e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:17 smithi087 podman[50450]: 2024-09-02 12:09:17.622019929 +0000 UTC m=+0.894434649 container cleanup e7448cffa948343e4a301e1f20b654f0c8a803e16431427a704f7a1acaefa35e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T12:09:17.750 INFO:journalctl@ceph.mon.b.smithi087.stdout:Sep 02 12:09:17 smithi087 bash[50450]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-b 2024-09-02T12:09:18.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b.service' 2024-09-02T12:09:18.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:18.091 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-02T12:09:18.091 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-02T12:09:18.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:18.398 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:18 smithi094 systemd[1]: Stopping Ceph mon.c for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:18.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-c[20388]: 2024-09-02T12:09:18.397+0000 7f0c961ce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:18.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-c[20388]: 2024-09-02T12:09:18.397+0000 7f0c961ce640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-02T12:09:18.732 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:18 smithi094 podman[51417]: 2024-09-02 12:09:18.489221442 +0000 UTC m=+0.207921483 container died 9b7c8f05dfb3be4c5d8aab86080b4e765b816a2f679a1112952d8b975717a5e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-02T12:09:19.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:18 smithi094 podman[51417]: 2024-09-02 12:09:18.889164159 +0000 UTC m=+0.607864194 container cleanup 9b7c8f05dfb3be4c5d8aab86080b4e765b816a2f679a1112952d8b975717a5e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731) 2024-09-02T12:09:19.096 INFO:journalctl@ceph.mon.c.smithi094.stdout:Sep 02 12:09:18 smithi094 bash[51417]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mon-c 2024-09-02T12:09:19.230 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c.service' 2024-09-02T12:09:19.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:19.285 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-02T12:09:19.285 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-02T12:09:19.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:19.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 12:09:19 smithi064 systemd[1]: Stopping Ceph mgr.x for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:19.844 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 12:09:19 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x[17390]: 2024-09-02T12:09:19.583+0000 7facd852e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:19.844 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 12:09:19 smithi064 podman[60234]: 2024-09-02 12:09:19.615089082 +0000 UTC m=+0.135429903 container died 2020f5ad642ae8a97212ebb4f9e48b9894276d8c1d45c45fd87497df0a1d40cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-02T12:09:20.344 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 12:09:19 smithi064 podman[60234]: 2024-09-02 12:09:19.96733594 +0000 UTC m=+0.487676748 container cleanup 2020f5ad642ae8a97212ebb4f9e48b9894276d8c1d45c45fd87497df0a1d40cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b59673c44bd569f9f3db37f87bced695dec5fcbf, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-02T12:09:20.345 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Sep 02 12:09:19 smithi064 bash[60234]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-mgr-x 2024-09-02T12:09:20.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x.service' 2024-09-02T12:09:20.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:20.725 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-02T12:09:20.726 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-02T12:09:20.726 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:21.094 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:20 smithi064 systemd[1]: Stopping Ceph osd.0 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:21.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:21.568 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0[28917]: 2024-09-02T12:09:21.130+0000 7fc2a62d0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:21.568 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0[28917]: 2024-09-02T12:09:21.130+0000 7fc2a62d0640 -1 osd.0 54 *** Got signal Terminated *** 2024-09-02T12:09:21.568 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:21 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0[28917]: 2024-09-02T12:09:21.130+0000 7fc2a62d0640 -1 osd.0 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:09:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:21.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:21.651 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:21.685 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:21.757 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:21.829 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:21.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:21.933 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:22.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:22.034 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:22.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:22.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:22.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-02T12:09:22.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~6s 2024-09-02T12:09:22.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-02T12:09:22.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-02T12:09:22.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-09-02T12:09:26.741 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:26 smithi064 podman[60357]: 2024-09-02 12:09:26.469577183 +0000 UTC m=+5.471810976 container died 5ffc06ce4035c822218d4f16fbf7961733a5e9d83fe9ebc20845a813074f2e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-nvmeof-b59673c) 2024-09-02T12:09:27.094 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:26 smithi064 podman[60357]: 2024-09-02 12:09:26.846814838 +0000 UTC m=+5.849048630 container cleanup 5ffc06ce4035c822218d4f16fbf7961733a5e9d83fe9ebc20845a813074f2e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-02T12:09:27.094 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:26 smithi064 bash[60357]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0 2024-09-02T12:09:27.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:27.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:27.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:27.271 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:27.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:27.417 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:27.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:27.491 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60453]: 2024-09-02 12:09:27.141624567 +0000 UTC m=+0.674788972 container remove 5ffc06ce4035c822218d4f16fbf7961733a5e9d83fe9ebc20845a813074f2e47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-nvmeof-b59673c, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-02T12:09:27.519 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:27.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:27.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:27.665 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:27.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:27.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~6s 2024-09-02T12:09:27.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~11s 2024-09-02T12:09:27.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2024-09-02T12:09:27.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2024-09-02T12:09:27.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~6s 2024-09-02T12:09:27.835 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60579]: 2024-09-02 12:09:27.394982515 +0000 UTC m=+0.017991720 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:09:27.836 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60579]: 2024-09-02 12:09:27.51569795 +0000 UTC m=+0.138707149 container create 3970267e9769f31273f7384fb17eda57dcc02fb742bdf4be0194f8f9d3b45761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:27.836 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60579]: 2024-09-02 12:09:27.718085165 +0000 UTC m=+0.341094367 container init 3970267e9769f31273f7384fb17eda57dcc02fb742bdf4be0194f8f9d3b45761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T12:09:27.836 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60579]: 2024-09-02 12:09:27.722607628 +0000 UTC m=+0.345616818 container start 3970267e9769f31273f7384fb17eda57dcc02fb742bdf4be0194f8f9d3b45761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-nvmeof-b59673c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-02T12:09:27.836 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60579]: 2024-09-02 12:09:27.834783804 +0000 UTC m=+0.457793003 container attach 3970267e9769f31273f7384fb17eda57dcc02fb742bdf4be0194f8f9d3b45761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:28.094 INFO:journalctl@ceph.osd.0.smithi064.stdout:Sep 02 12:09:27 smithi064 podman[60579]: 2024-09-02 12:09:27.917504445 +0000 UTC m=+0.540513649 container died 3970267e9769f31273f7384fb17eda57dcc02fb742bdf4be0194f8f9d3b45761 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-nvmeof-b59673c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T12:09:28.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0.service' 2024-09-02T12:09:28.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:28.573 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-02T12:09:28.574 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-02T12:09:28.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:28.809 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:28 smithi064 systemd[1]: Stopping Ceph osd.1 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:29.094 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1[34812]: 2024-09-02T12:09:28.948+0000 7fb4e9c40640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:29.094 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1[34812]: 2024-09-02T12:09:28.948+0000 7fb4e9c40640 -1 osd.1 54 *** Got signal Terminated *** 2024-09-02T12:09:29.095 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:28 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1[34812]: 2024-09-02T12:09:28.948+0000 7fb4e9c40640 -1 osd.1 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:09:32.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:32.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:32.835 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:32.868 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:32.940 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:33.013 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:33.046 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:33.117 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:33.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:33.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:33.249 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:33.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:33.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-09-02T12:09:33.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-02T12:09:33.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~17s 2024-09-02T12:09:33.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~11s 2024-09-02T12:09:33.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~11s 2024-09-02T12:09:33.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~11s 2024-09-02T12:09:34.717 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:34 smithi064 podman[60721]: 2024-09-02 12:09:34.454187501 +0000 UTC m=+5.628029088 container died c2fc1b33e96ff9f54d7ff5b6d0f59c7966a9cdf65364a93669acacf27820f9f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-02T12:09:35.094 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:34 smithi064 podman[60721]: 2024-09-02 12:09:34.81584533 +0000 UTC m=+5.989686919 container cleanup c2fc1b33e96ff9f54d7ff5b6d0f59c7966a9cdf65364a93669acacf27820f9f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2) 2024-09-02T12:09:35.095 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:34 smithi064 bash[60721]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1 2024-09-02T12:09:35.475 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60828]: 2024-09-02 12:09:35.141968697 +0000 UTC m=+0.686414919 container remove c2fc1b33e96ff9f54d7ff5b6d0f59c7966a9cdf65364a93669acacf27820f9f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-02T12:09:35.779 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60921]: 2024-09-02 12:09:35.377166617 +0000 UTC m=+0.016687146 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:09:35.780 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60921]: 2024-09-02 12:09:35.476007943 +0000 UTC m=+0.115528464 container create 9d0cc65a223b6c92255a3039f4a62ac75cf352627c7e07b4b3db802ba0bdd1a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-02T12:09:35.780 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60921]: 2024-09-02 12:09:35.695610133 +0000 UTC m=+0.335130655 container init 9d0cc65a223b6c92255a3039f4a62ac75cf352627c7e07b4b3db802ba0bdd1a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T12:09:35.780 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60921]: 2024-09-02 12:09:35.698762656 +0000 UTC m=+0.338283187 container start 9d0cc65a223b6c92255a3039f4a62ac75cf352627c7e07b4b3db802ba0bdd1a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-nvmeof-b59673c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True) 2024-09-02T12:09:35.780 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60921]: 2024-09-02 12:09:35.778950345 +0000 UTC m=+0.418470873 container attach 9d0cc65a223b6c92255a3039f4a62ac75cf352627c7e07b4b3db802ba0bdd1a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:36.094 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:35 smithi064 podman[60921]: 2024-09-02 12:09:35.907118224 +0000 UTC m=+0.546638746 container died 9d0cc65a223b6c92255a3039f4a62ac75cf352627c7e07b4b3db802ba0bdd1a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-02T12:09:36.441 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:36 smithi064 podman[60944]: 2024-09-02 12:09:36.423625998 +0000 UTC m=+0.508189685 container remove 9d0cc65a223b6c92255a3039f4a62ac75cf352627c7e07b4b3db802ba0bdd1a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-1-deactivate, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:36.441 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:36 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-02T12:09:36.441 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:36 smithi064 systemd[1]: Stopped Ceph osd.1 for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:09:36.441 INFO:journalctl@ceph.osd.1.smithi064.stdout:Sep 02 12:09:36 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1.service: Consumed 5.578s CPU time. 2024-09-02T12:09:36.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1.service' 2024-09-02T12:09:36.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:36.479 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-02T12:09:36.480 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-02T12:09:36.480 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:36.785 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:36 smithi087 systemd[1]: Stopping Ceph osd.2 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:37.250 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:36 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2[24647]: 2024-09-02T12:09:36.783+0000 7ff9827d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:37.250 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:36 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2[24647]: 2024-09-02T12:09:36.783+0000 7ff9827d9640 -1 osd.2 54 *** Got signal Terminated *** 2024-09-02T12:09:37.250 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:36 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2[24647]: 2024-09-02T12:09:36.783+0000 7ff9827d9640 -1 osd.2 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:09:38.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:38.519 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:38.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:38.623 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:38.695 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:38.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.828 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:38.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~17s 2024-09-02T12:09:38.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2024-09-02T12:09:38.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-02T12:09:38.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~22s 2024-09-02T12:09:38.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~17s 2024-09-02T12:09:38.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~17s 2024-09-02T12:09:38.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~17s 2024-09-02T12:09:42.584 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:42 smithi087 podman[50833]: 2024-09-02 12:09:42.26546127 +0000 UTC m=+5.594132938 container died 277c5fcec278fd04a9fba60898ed2e62e6911613dffa2d3e6272b5a48a405ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-02T12:09:42.584 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:42 smithi087 podman[50833]: 2024-09-02 12:09:42.583562547 +0000 UTC m=+5.912234215 container cleanup 277c5fcec278fd04a9fba60898ed2e62e6911613dffa2d3e6272b5a48a405ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:43.000 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:42 smithi087 bash[50833]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2 2024-09-02T12:09:43.000 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:42 smithi087 podman[50930]: 2024-09-02 12:09:42.785107306 +0000 UTC m=+0.523079967 container remove 277c5fcec278fd04a9fba60898ed2e62e6911613dffa2d3e6272b5a48a405ec1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-02T12:09:43.455 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:43 smithi087 podman[51022]: 2024-09-02 12:09:43.09732438 +0000 UTC m=+0.017252773 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:09:43.456 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:43 smithi087 podman[51022]: 2024-09-02 12:09:43.2025546 +0000 UTC m=+0.122482988 container create 3a64d2ee24378d8b6a72d57232344cc6ccdf4482c39a84f24608d067b1617e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-02T12:09:43.456 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:43 smithi087 podman[51022]: 2024-09-02 12:09:43.455091509 +0000 UTC m=+0.375019893 container init 3a64d2ee24378d8b6a72d57232344cc6ccdf4482c39a84f24608d067b1617e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-02T12:09:43.750 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:43 smithi087 podman[51022]: 2024-09-02 12:09:43.458772975 +0000 UTC m=+0.378701357 container start 3a64d2ee24378d8b6a72d57232344cc6ccdf4482c39a84f24608d067b1617e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T12:09:43.750 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:43 smithi087 podman[51022]: 2024-09-02 12:09:43.563705699 +0000 UTC m=+0.483634097 container attach 3a64d2ee24378d8b6a72d57232344cc6ccdf4482c39a84f24608d067b1617e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731) 2024-09-02T12:09:43.751 INFO:journalctl@ceph.osd.2.smithi087.stdout:Sep 02 12:09:43 smithi087 podman[51022]: 2024-09-02 12:09:43.657674951 +0000 UTC m=+0.577603340 container died 3a64d2ee24378d8b6a72d57232344cc6ccdf4482c39a84f24608d067b1617e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-02T12:09:43.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:43.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:43.991 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:44.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:44.107 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:44.180 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:44.212 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:44.282 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:44.337 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2.service' 2024-09-02T12:09:44.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:44.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:44.373 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-02T12:09:44.374 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-02T12:09:44.374 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:44.385 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:44.451 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:44.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:44.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2024-09-02T12:09:44.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-09-02T12:09:44.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2024-09-02T12:09:44.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~28s 2024-09-02T12:09:44.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~22s 2024-09-02T12:09:44.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~22s 2024-09-02T12:09:44.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~22s 2024-09-02T12:09:44.748 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:44 smithi087 systemd[1]: Stopping Ceph osd.3 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:45.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3[30514]: 2024-09-02T12:09:44.746+0000 7f3a9b4e9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:45.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3[30514]: 2024-09-02T12:09:44.746+0000 7f3a9b4e9640 -1 osd.3 54 *** Got signal Terminated *** 2024-09-02T12:09:45.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:44 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3[30514]: 2024-09-02T12:09:44.746+0000 7f3a9b4e9640 -1 osd.3 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:09:47.536 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:47.151+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:47.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:47 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:47.535+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:48.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:48 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:48.186+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:48.557 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:48 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:48.156+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:48.557 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:48 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:48.323+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:48.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:48 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:48.555+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:49.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:49 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:49.195+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:49.514 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:49.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:49.564 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:49.112+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:49.564 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:49.329+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:49.615 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:49.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:49.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:49.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:49.815 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:49 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:49.563+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:49.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:49.896 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:49.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:50.000 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:50.032 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:50.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:50.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~28s 2024-09-02T12:09:50.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~17s 2024-09-02T12:09:50.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2024-09-02T12:09:50.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-02T12:09:50.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~34s 2024-09-02T12:09:50.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~28s 2024-09-02T12:09:50.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~28s 2024-09-02T12:09:50.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~28s 2024-09-02T12:09:50.273 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:50 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:50.192+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:50.344 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:50 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:50.075+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:50.564 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:50 smithi087 podman[51208]: 2024-09-02 12:09:50.273155841 +0000 UTC m=+5.662372348 container died d2e243a77dd38c419a5b3caf4aa18121bd0354946e491c45147f72609c4ce95d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-nvmeof-b59673c, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-02T12:09:50.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:50 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:50.343+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:50.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:50 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:50.551+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:50.914 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:50 smithi087 podman[51208]: 2024-09-02 12:09:50.687409231 +0000 UTC m=+6.076625731 container cleanup d2e243a77dd38c419a5b3caf4aa18121bd0354946e491c45147f72609c4ce95d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-02T12:09:50.914 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:50 smithi087 bash[51208]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3 2024-09-02T12:09:51.181 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:50 smithi087 podman[51307]: 2024-09-02 12:09:50.913583404 +0000 UTC m=+0.635083368 container remove d2e243a77dd38c419a5b3caf4aa18121bd0354946e491c45147f72609c4ce95d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-02T12:09:51.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:51.297+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:51.346 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:51.070+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:51.432 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:51 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:51.179+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:51.433 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:51 smithi087 podman[51394]: 2024-09-02 12:09:51.161673106 +0000 UTC m=+0.021615783 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:09:51.433 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:51 smithi087 podman[51394]: 2024-09-02 12:09:51.296359372 +0000 UTC m=+0.156302042 container create 86a1ba8ffba3fe0b528bbe9786bc4beca80a5610b09b98b812696a87af004058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-02T12:09:51.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:51.530+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:52.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:51 smithi087 podman[51394]: 2024-09-02 12:09:51.525718254 +0000 UTC m=+0.385660933 container init 86a1ba8ffba3fe0b528bbe9786bc4beca80a5610b09b98b812696a87af004058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-02T12:09:52.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:51 smithi087 podman[51394]: 2024-09-02 12:09:51.529404636 +0000 UTC m=+0.389347304 container start 86a1ba8ffba3fe0b528bbe9786bc4beca80a5610b09b98b812696a87af004058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3-deactivate, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-02T12:09:52.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:51 smithi087 podman[51394]: 2024-09-02 12:09:51.592142721 +0000 UTC m=+0.452085388 container attach 86a1ba8ffba3fe0b528bbe9786bc4beca80a5610b09b98b812696a87af004058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-nvmeof-b59673c, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T12:09:52.000 INFO:journalctl@ceph.osd.3.smithi087.stdout:Sep 02 12:09:51 smithi087 podman[51394]: 2024-09-02 12:09:51.727517068 +0000 UTC m=+0.587459738 container died 86a1ba8ffba3fe0b528bbe9786bc4beca80a5610b09b98b812696a87af004058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-02T12:09:52.279 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:52.153+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:52.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3.service' 2024-09-02T12:09:52.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:09:52.333 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-02T12:09:52.333 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-02T12:09:52.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:52.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:52.308+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:52.346 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:52.085+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:52.699 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:52 smithi087 systemd[1]: Stopping Ceph osd.4 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:09:52.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:52 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:52.495+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:53.000 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:52.696+0000 7fa95118f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:09:53.000 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:52.696+0000 7fa95118f640 -1 osd.4 54 *** Got signal Terminated *** 2024-09-02T12:09:53.000 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:52 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:52.696+0000 7fa95118f640 -1 osd.4 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:09:53.346 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:53 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:53.059+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:53.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:53 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:53.172+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:53.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:53 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:53.501+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:53.846 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:53 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:53.345+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:54.346 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:54.024+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:54.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:54 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:54.155+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:54.846 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:54.349+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:54.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:54 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:54.517+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:55.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:09:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:09:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.200 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:09:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.233 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:09:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:09:55.346 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:55 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:55.008+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.378 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:09:55.409 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:09:55.479 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:09:55.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:55 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:55.201+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:55.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:09:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:09:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.613 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:09:55.634 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:55 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:55.558+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:55.634 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:55 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:55.558+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:09:55.634 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:55 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:55.373+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:55.634 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:55 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:55.373+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:09:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:09:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:09:55.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~34s 2024-09-02T12:09:55.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-09-02T12:09:55.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~17s 2024-09-02T12:09:55.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2024-09-02T12:09:55.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-02T12:09:55.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~39s 2024-09-02T12:09:55.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~34s 2024-09-02T12:09:55.675 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~34s 2024-09-02T12:09:55.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~34s 2024-09-02T12:09:56.337 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:56.008+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:56.337 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:56.008+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:09:56.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:56.235+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:56.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:56 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:56.235+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:30.804221+0000 front 2024-09-02T12:09:30.804235+0000 (oldest deadline 2024-09-02T12:09:56.104297+0000) 2024-09-02T12:09:56.594 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:56.335+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:56.594 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:56.335+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:09:56.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:56.593+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:56.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:56.593+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:09:57.322 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:56.990+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:57.323 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:56 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:56.990+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:09:57.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:57 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:57.321+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:57.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:57 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:57.321+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:09:57.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:57 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:57.544+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:57.597 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:57 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:57.544+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:09:57.666 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:57 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:57.256+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.203739+0000 front 2024-09-02T12:09:23.203676+0000 (oldest deadline 2024-09-02T12:09:47.303720+0000) 2024-09-02T12:09:57.666 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:57 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4[36028]: 2024-09-02T12:09:57.256+0000 7fa94979c640 -1 osd.4 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:30.804221+0000 front 2024-09-02T12:09:30.804235+0000 (oldest deadline 2024-09-02T12:09:56.104297+0000) 2024-09-02T12:09:58.315 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:57 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:57.943+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:58.316 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:57 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:57.943+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:09:58.500 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:58 smithi087 podman[51498]: 2024-09-02 12:09:58.236716772 +0000 UTC m=+5.647920907 container died a4c97f18837f1325d4079a43a5b9ce1cdcde1af82455d037e3ad69c95971bd96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-nvmeof-b59673c) 2024-09-02T12:09:58.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:58.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:58.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:58.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:09:58.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:58.551+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:58.597 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:58.551+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:09:58.909 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:58 smithi087 podman[51498]: 2024-09-02 12:09:58.597835626 +0000 UTC m=+6.009039752 container cleanup a4c97f18837f1325d4079a43a5b9ce1cdcde1af82455d037e3ad69c95971bd96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4, RELEASE=main-nvmeof-b59673c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-02T12:09:58.909 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:58 smithi087 bash[51498]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4 2024-09-02T12:09:59.250 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:58 smithi087 podman[51605]: 2024-09-02 12:09:58.907258884 +0000 UTC m=+0.670007100 container remove a4c97f18837f1325d4079a43a5b9ce1cdcde1af82455d037e3ad69c95971bd96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T12:09:59.266 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:58.924+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:09:59.266 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:58 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:58.924+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:09:59.530 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:59.511+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:09:59.530 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:09:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:09:59.511+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:09:59.530 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:59.265+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:09:59.530 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:09:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:09:59.265+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:09:59.578 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:59 smithi087 podman[51690]: 2024-09-02 12:09:59.177330681 +0000 UTC m=+0.016387764 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:09:59.578 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:59 smithi087 podman[51690]: 2024-09-02 12:09:59.283358084 +0000 UTC m=+0.122415165 container create 850be8cc6ffbef96076ffc556e5fc5b0773176da61dae7307296e53d15247eb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-02T12:09:59.578 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:59 smithi087 podman[51690]: 2024-09-02 12:09:59.502334391 +0000 UTC m=+0.341391466 container init 850be8cc6ffbef96076ffc556e5fc5b0773176da61dae7307296e53d15247eb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=main-nvmeof-b59673c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-02T12:09:59.579 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:59 smithi087 podman[51690]: 2024-09-02 12:09:59.505551051 +0000 UTC m=+0.344608131 container start 850be8cc6ffbef96076ffc556e5fc5b0773176da61dae7307296e53d15247eb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-nvmeof-b59673c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T12:09:59.579 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:59 smithi087 podman[51690]: 2024-09-02 12:09:59.577349935 +0000 UTC m=+0.416407023 container attach 850be8cc6ffbef96076ffc556e5fc5b0773176da61dae7307296e53d15247eb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T12:09:59.989 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:09:59 smithi087 podman[51690]: 2024-09-02 12:09:59.702619977 +0000 UTC m=+0.541677061 container died 850be8cc6ffbef96076ffc556e5fc5b0773176da61dae7307296e53d15247eb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-nvmeof-b59673c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-09-02T12:10:00.219 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4.service' 2024-09-02T12:10:00.242 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:10:00 smithi087 podman[51716]: 2024-09-02 12:10:00.197041889 +0000 UTC m=+0.488254648 container remove 850be8cc6ffbef96076ffc556e5fc5b0773176da61dae7307296e53d15247eb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-02T12:10:00.242 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:10:00 smithi087 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4.service: Deactivated successfully. 2024-09-02T12:10:00.242 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:10:00 smithi087 systemd[1]: Stopped Ceph osd.4 for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:10:00.242 INFO:journalctl@ceph.osd.4.smithi087.stdout:Sep 02 12:10:00 smithi087 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4.service: Consumed 3.883s CPU time. 2024-09-02T12:10:00.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:10:00.256 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-02T12:10:00.257 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-02T12:10:00.257 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:00.259 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:59.939+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:10:00.260 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:09:59 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:09:59.939+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:10:00.515 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:00 smithi094 systemd[1]: Stopping Ceph osd.5 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:10:00.515 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:00.257+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:00.515 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:00.257+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:00.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:00.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:00.780 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:00.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:00.846 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:00.601+0000 7f749699b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:10:00.846 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:00.601+0000 7f749699b640 -1 osd.5 54 *** Got signal Terminated *** 2024-09-02T12:10:00.846 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:00.601+0000 7f749699b640 -1 osd.5 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:10:00.847 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:00.551+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:00.847 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:00.551+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:00.882 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:00.952 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:00.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:01.057 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:01.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:01.161 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:01.191 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:01.220 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:00.964+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:10:01.220 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:00 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:00.964+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:10:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:01.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:01.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~39s 2024-09-02T12:10:01.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~28s 2024-09-02T12:10:01.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2024-09-02T12:10:01.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s 2024-09-02T12:10:01.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2024-09-02T12:10:01.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-02T12:10:01.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~45s 2024-09-02T12:10:01.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~39s 2024-09-02T12:10:01.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~39s 2024-09-02T12:10:01.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~39s 2024-09-02T12:10:01.507 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:01.305+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:01.507 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:01.305+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:01.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:01.506+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:01.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:01.506+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:02.316 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:01.934+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:10:02.316 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:01 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:01.934+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:10:02.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:02.504+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:02.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:02.504+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:02.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:02.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:02.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:02.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:03.266 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:02.976+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:10:03.267 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:02 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:02.976+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:10:03.531 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:03.265+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:03.531 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:03.265+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:03.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:03.530+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:03.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:03.530+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:04.309 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:03.964+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:10:04.309 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:03.964+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:10:04.309 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:03 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:03.964+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.285538+0000 front 2024-09-02T12:09:41.285591+0000 (oldest deadline 2024-09-02T12:10:03.585461+0000) 2024-09-02T12:10:04.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:04.555+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:04.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:04.555+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:04.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:04.307+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:04.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:04.307+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:05.305 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:04.999+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:25.683875+0000 front 2024-09-02T12:09:25.683886+0000 (oldest deadline 2024-09-02T12:09:46.783798+0000) 2024-09-02T12:10:05.305 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:04.999+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.484862+0000 front 2024-09-02T12:09:32.484911+0000 (oldest deadline 2024-09-02T12:09:55.384919+0000) 2024-09-02T12:10:05.305 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:04 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5[24385]: 2024-09-02T12:10:04.999+0000 7f748efa8640 -1 osd.5 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.285538+0000 front 2024-09-02T12:09:41.285591+0000 (oldest deadline 2024-09-02T12:10:03.585461+0000) 2024-09-02T12:10:05.589 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:05.303+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:05.589 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:05.303+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:05.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:05.587+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:05.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:05.587+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:05.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:05 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:05.587+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:06.236 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:05 smithi094 podman[52130]: 2024-09-02 12:10:05.969744367 +0000 UTC m=+5.496751236 container died 53daeb17158c085b68d757354a70507fc36fc8f1c9710b45e8866a5fa5f66a00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, org.label-schema.license=GPLv2) 2024-09-02T12:10:06.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.358 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.390 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.460 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.530 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:06.552 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:06 smithi094 podman[52130]: 2024-09-02 12:10:06.335075642 +0000 UTC m=+5.862082503 container cleanup 53daeb17158c085b68d757354a70507fc36fc8f1c9710b45e8866a5fa5f66a00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T12:10:06.552 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:06 smithi094 bash[52130]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5 2024-09-02T12:10:06.553 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:06.262+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:06.553 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:06.262+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:06.553 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:06.262+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.564 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:06.636 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:06.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.740 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.770 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:06.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~45s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~34s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~28s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~17s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~11s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~50s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~45s 2024-09-02T12:10:06.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~45s 2024-09-02T12:10:06.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~45s 2024-09-02T12:10:06.846 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:06 smithi094 podman[52223]: 2024-09-02 12:10:06.636674492 +0000 UTC m=+0.668370020 container remove 53daeb17158c085b68d757354a70507fc36fc8f1c9710b45e8866a5fa5f66a00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux ) 2024-09-02T12:10:06.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:06.628+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:06.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:06.628+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:06.847 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:06 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:06.628+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:07.277 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:06 smithi094 podman[52389]: 2024-09-02 12:10:06.899551932 +0000 UTC m=+0.016077891 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:10:07.278 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:07 smithi094 podman[52389]: 2024-09-02 12:10:07.021244428 +0000 UTC m=+0.137770389 container create 81b58f7fa7e339b7c6c7f9db4a1148f459754cee67372061f509fbedb2919c27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-02T12:10:07.278 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:07 smithi094 podman[52389]: 2024-09-02 12:10:07.223462992 +0000 UTC m=+0.339988951 container init 81b58f7fa7e339b7c6c7f9db4a1148f459754cee67372061f509fbedb2919c27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-02T12:10:07.278 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:07 smithi094 podman[52389]: 2024-09-02 12:10:07.226712197 +0000 UTC m=+0.343238158 container start 81b58f7fa7e339b7c6c7f9db4a1148f459754cee67372061f509fbedb2919c27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-02T12:10:07.585 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:07 smithi094 podman[52389]: 2024-09-02 12:10:07.348887123 +0000 UTC m=+0.465413089 container attach 81b58f7fa7e339b7c6c7f9db4a1148f459754cee67372061f509fbedb2919c27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-02T12:10:07.586 INFO:journalctl@ceph.osd.5.smithi094.stdout:Sep 02 12:10:07 smithi094 podman[52389]: 2024-09-02 12:10:07.417845951 +0000 UTC m=+0.534371907 container died 81b58f7fa7e339b7c6c7f9db4a1148f459754cee67372061f509fbedb2919c27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-nvmeof-b59673c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-02T12:10:07.586 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:07.276+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:07.586 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:07.276+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:07.586 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:07.276+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:07.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:07.583+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:07.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:07.583+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:07.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:07 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:07.583+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:07.964 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5.service' 2024-09-02T12:10:08.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:10:08.001 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-02T12:10:08.001 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-02T12:10:08.001 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:08.297 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 systemd[1]: Stopping Ceph osd.6 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:10:08.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:08.295+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:08.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:08.295+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:08.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:08.295+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:08.597 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:08.363+0000 7fd759206640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:10:08.597 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:08.363+0000 7fd759206640 -1 osd.6 54 *** Got signal Terminated *** 2024-09-02T12:10:08.597 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:08.363+0000 7fd759206640 -1 osd.6 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:10:09.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:08.603+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:09.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:08.603+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:09.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:08 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:08.603+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:09.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:09.308+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:09.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:09.308+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:09.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:09.308+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:10.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:09.617+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:10.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:09.617+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:10.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:09 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:09.617+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:10.588 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:10.297+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:10.588 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:10.297+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:10.588 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:10.297+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:10.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:10.585+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:10.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:10.585+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:10.846 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:10 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:10.585+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:11.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:11.303+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:11.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:11.303+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:11.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:11.303+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:11.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:11.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:11.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:11.968 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:12.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:11.620+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:12.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:11.620+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:12.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:11 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:11.620+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.142 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.214 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:12.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.315 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.345 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~50s 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~39s 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~34s 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~22s 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~17s 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~11s 2024-09-02T12:10:12.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-09-02T12:10:12.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~56s 2024-09-02T12:10:12.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~50s 2024-09-02T12:10:12.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~50s 2024-09-02T12:10:12.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~50s 2024-09-02T12:10:12.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:12.353+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:12.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:12.353+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:12.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:12.353+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:13.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:12.623+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:22.909632+0000 front 2024-09-02T12:09:22.909685+0000 (oldest deadline 2024-09-02T12:09:47.009687+0000) 2024-09-02T12:10:13.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:12.623+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:29.910207+0000 front 2024-09-02T12:09:29.910181+0000 (oldest deadline 2024-09-02T12:09:54.610233+0000) 2024-09-02T12:10:13.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:12 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6[30191]: 2024-09-02T12:10:12.623+0000 7fd751813640 -1 osd.6 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.011189+0000 front 2024-09-02T12:09:41.011150+0000 (oldest deadline 2024-09-02T12:10:05.111050+0000) 2024-09-02T12:10:13.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:13.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:13.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:13.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:13.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:13.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:13.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:13 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:13.314+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:14.041 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:13 smithi094 podman[52492]: 2024-09-02 12:10:13.725819706 +0000 UTC m=+5.474894771 container died a21034e7f87c757903cee1f600b4b77c5ac2d38ae7364a9cbb8087cafa729e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=main-nvmeof-b59673c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-02T12:10:14.343 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:14 smithi094 podman[52492]: 2024-09-02 12:10:14.147188747 +0000 UTC m=+5.896263807 container cleanup a21034e7f87c757903cee1f600b4b77c5ac2d38ae7364a9cbb8087cafa729e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-02T12:10:14.344 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:14 smithi094 bash[52492]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6 2024-09-02T12:10:14.596 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:14 smithi094 podman[52589]: 2024-09-02 12:10:14.406940584 +0000 UTC m=+0.679773895 container remove a21034e7f87c757903cee1f600b4b77c5ac2d38ae7364a9cbb8087cafa729e0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c) 2024-09-02T12:10:14.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:14.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:14.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:14.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:14.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:14.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:14.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:14 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:14.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:15.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:14 smithi094 podman[52681]: 2024-09-02 12:10:14.73835875 +0000 UTC m=+0.015713083 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:10:15.096 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:14 smithi094 podman[52681]: 2024-09-02 12:10:14.857866379 +0000 UTC m=+0.135220695 container create 4510c91ad14ec8b517f69674e3cb304f21fda735af0da54b31344520cdea8963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-02T12:10:15.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:15.334+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:15.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:15.334+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:15.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:15.334+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:15.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:15 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:15.334+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:15.597 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:15 smithi094 podman[52681]: 2024-09-02 12:10:15.104419953 +0000 UTC m=+0.381774280 container init 4510c91ad14ec8b517f69674e3cb304f21fda735af0da54b31344520cdea8963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-02T12:10:15.598 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:15 smithi094 podman[52681]: 2024-09-02 12:10:15.107810125 +0000 UTC m=+0.385164451 container start 4510c91ad14ec8b517f69674e3cb304f21fda735af0da54b31344520cdea8963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-nvmeof-b59673c, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-02T12:10:15.598 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:15 smithi094 podman[52681]: 2024-09-02 12:10:15.212891306 +0000 UTC m=+0.490245624 container attach 4510c91ad14ec8b517f69674e3cb304f21fda735af0da54b31344520cdea8963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-nvmeof-b59673c, GIT_CLEAN=True) 2024-09-02T12:10:15.598 INFO:journalctl@ceph.osd.6.smithi094.stdout:Sep 02 12:10:15 smithi094 podman[52681]: 2024-09-02 12:10:15.298358634 +0000 UTC m=+0.575712954 container died 4510c91ad14ec8b517f69674e3cb304f21fda735af0da54b31344520cdea8963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-02T12:10:16.009 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6.service' 2024-09-02T12:10:16.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:10:16.046 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-02T12:10:16.047 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-02T12:10:16.047 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:16.329 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 systemd[1]: Stopping Ceph osd.7 for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:10:16.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.327+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.327+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.327+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.327+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.327+0000 7f9e81bd8640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15144.0:1127 2.10 2:0c6869da:::rbd_header.3a38cc8cd89c:head [watch ping cookie 140067607899872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.403+0000 7f9e855c3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.403+0000 7f9e855c3640 -1 osd.7 54 *** Got signal Terminated *** 2024-09-02T12:10:16.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:16 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:16.403+0000 7f9e855c3640 -1 osd.7 54 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T12:10:17.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.514 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.545 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:17.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:17.286+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:17.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:17.286+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:17.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:17.286+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:17.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:17.286+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:17.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:17 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:17.286+0000 7f9e81bd8640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15144.0:1127 2.10 2:0c6869da:::rbd_header.3a38cc8cd89c:head [watch ping cookie 140067607899872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) 2024-09-02T12:10:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.615 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.685 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.717 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.789 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.923 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:17.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~56s 2024-09-02T12:10:17.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~45s 2024-09-02T12:10:17.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~39s 2024-09-02T12:10:17.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~28s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~22s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~17s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~61s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~56s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~56s 2024-09-02T12:10:17.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~56s 2024-09-02T12:10:18.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:18.292+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:18.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:18.292+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:18.347 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:18.292+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:18.347 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:18.292+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:18.347 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:18 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:18.292+0000 7f9e81bd8640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15144.0:1127 2.10 2:0c6869da:::rbd_header.3a38cc8cd89c:head [watch ping cookie 140067607899872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) 2024-09-02T12:10:19.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:19 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:19.322+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:19.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:19 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:19.322+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:19.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:19 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:19.322+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:19.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:19 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:19.322+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:19.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:19 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:19.322+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6820 osd.4 since back 2024-09-02T12:09:53.966873+0000 front 2024-09-02T12:09:53.966830+0000 (oldest deadline 2024-09-02T12:10:18.666795+0000) 2024-09-02T12:10:19.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:19 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:19.322+0000 7f9e81bd8640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15144.0:1127 2.10 2:0c6869da:::rbd_header.3a38cc8cd89c:head [watch ping cookie 140067607899872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) 2024-09-02T12:10:20.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:20.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:20.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:20.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:20.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:20.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:20.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:20.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:20.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:20.342+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6820 osd.4 since back 2024-09-02T12:09:53.966873+0000 front 2024-09-02T12:09:53.966830+0000 (oldest deadline 2024-09-02T12:10:18.666795+0000) 2024-09-02T12:10:20.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:20 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:20.342+0000 7f9e81bd8640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15144.0:1127 2.10 2:0c6869da:::rbd_header.3a38cc8cd89c:head [watch ping cookie 140067607899872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) 2024-09-02T12:10:21.596 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:21.324+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-09-02T12:09:23.963537+0000 front 2024-09-02T12:09:23.963520+0000 (oldest deadline 2024-09-02T12:09:48.063553+0000) 2024-09-02T12:10:21.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:21.324+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-09-02T12:09:32.164093+0000 front 2024-09-02T12:09:32.164073+0000 (oldest deadline 2024-09-02T12:09:55.064107+0000) 2024-09-02T12:10:21.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:21.324+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6804 osd.2 since back 2024-09-02T12:09:41.165916+0000 front 2024-09-02T12:09:41.165848+0000 (oldest deadline 2024-09-02T12:10:05.865746+0000) 2024-09-02T12:10:21.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:21.324+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6812 osd.3 since back 2024-09-02T12:09:45.866090+0000 front 2024-09-02T12:09:45.866036+0000 (oldest deadline 2024-09-02T12:10:12.866298+0000) 2024-09-02T12:10:21.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:21.324+0000 7f9e81bd8640 -1 osd.7 54 heartbeat_check: no reply from 172.21.15.87:6820 osd.4 since back 2024-09-02T12:09:53.966873+0000 front 2024-09-02T12:09:53.966830+0000 (oldest deadline 2024-09-02T12:10:18.666795+0000) 2024-09-02T12:10:21.597 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7[35700]: 2024-09-02T12:10:21.324+0000 7f9e81bd8640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15144.0:1127 2.10 2:0c6869da:::rbd_header.3a38cc8cd89c:head [watch ping cookie 140067607899872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) 2024-09-02T12:10:22.005 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:21 smithi094 podman[52784]: 2024-09-02 12:10:21.7370328 +0000 UTC m=+5.470839037 container died 49beed78224de6899b0de0ae133e011d1b776213617765550026bc8de168bf50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=main-nvmeof-b59673c, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T12:10:22.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 podman[52784]: 2024-09-02 12:10:22.103839873 +0000 UTC m=+5.837646115 container cleanup 49beed78224de6899b0de0ae133e011d1b776213617765550026bc8de168bf50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-nvmeof-b59673c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T12:10:22.346 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 bash[52784]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7 2024-09-02T12:10:22.717 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 podman[52884]: 2024-09-02 12:10:22.374788509 +0000 UTC m=+0.635985387 container remove 49beed78224de6899b0de0ae133e011d1b776213617765550026bc8de168bf50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-nvmeof-b59673c, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-02T12:10:22.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:23.034 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 podman[52980]: 2024-09-02 12:10:22.619061504 +0000 UTC m=+0.015668082 image pull b92d9064031cea4134dd32390f6904a4b1e8523bd2a4bbb61f509e9ff7e02238 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81 2024-09-02T12:10:23.034 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 podman[52980]: 2024-09-02 12:10:22.780999101 +0000 UTC m=+0.177605675 container create 6901a5b2c10faa7256a1c0362365a81ef3700500e3b9f3db97e2806016f5fb5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-02T12:10:23.034 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 podman[52980]: 2024-09-02 12:10:22.958311361 +0000 UTC m=+0.354917934 container init 6901a5b2c10faa7256a1c0362365a81ef3700500e3b9f3db97e2806016f5fb5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7-deactivate, RELEASE=main-nvmeof-b59673c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-02T12:10:23.035 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:22 smithi094 podman[52980]: 2024-09-02 12:10:22.962517302 +0000 UTC m=+0.359123883 container start 6901a5b2c10faa7256a1c0362365a81ef3700500e3b9f3db97e2806016f5fb5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-nvmeof-b59673c, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-02T12:10:23.035 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:23 smithi094 podman[52980]: 2024-09-02 12:10:23.033373531 +0000 UTC m=+0.429980113 container attach 6901a5b2c10faa7256a1c0362365a81ef3700500e3b9f3db97e2806016f5fb5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-nvmeof-b59673c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-02T12:10:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.119 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.190 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:23.261 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.261 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:23.285 INFO:journalctl@ceph.osd.7.smithi094.stdout:Sep 02 12:10:23 smithi094 podman[52980]: 2024-09-02 12:10:23.152826105 +0000 UTC m=+0.549432681 container died 6901a5b2c10faa7256a1c0362365a81ef3700500e3b9f3db97e2806016f5fb5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:947508d658b3c61f07b952b0a838c7fb40b939364cc76dcfb4f16b4babc84d81, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=main-nvmeof-b59673c, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T12:10:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.367 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.471 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.502 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~61s 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~45s 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~33s 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~28s 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~22s 2024-09-02T12:10:23.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~11s 2024-09-02T12:10:23.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~6s 2024-09-02T12:10:23.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~67s 2024-09-02T12:10:23.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~61s 2024-09-02T12:10:23.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~61s 2024-09-02T12:10:23.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~61s 2024-09-02T12:10:23.782 DEBUG:teuthology.orchestra.run.smithi094:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7.service' 2024-09-02T12:10:23.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T12:10:23.818 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-02T12:10:23.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T12:10:24.029 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:10:24.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T12:10:24.628 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:10:25.718 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 systemd[1]: Stopping Ceph nvmeof.nvmeof.a for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:10:26.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:10:25] ERROR server.py:111 (2): GatewayServer exception occurred: 2024-09-02T12:10:26.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: Traceback (most recent call last): 2024-09-02T12:10:26.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/src/control/__main__.py", line 39, in 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: gateway.keep_alive() 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/src/control/server.py", line 526, in keep_alive 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: timedout = self.server.wait_for_termination(timeout=1) 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/src/__pypackages__/3.9/lib/grpc/_server.py", line 1118, in wait_for_termination 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: return _common.wait(self._state.termination_event.wait, 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 157, in wait 2024-09-02T12:10:26.096 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: _wait_once(wait_fn, remaining, spin_cb) 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 112, in _wait_once 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: wait_fn(timeout=timeout) 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/usr/lib64/python3.9/threading.py", line 581, in wait 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: signaled = self._cond.wait(timeout) 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/usr/lib64/python3.9/threading.py", line 316, in wait 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: gotit = waiter.acquire(True, timeout) 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: File "/src/control/__main__.py", line 17, in sigterm_handler 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: raise SystemExit(f"Gateway process terminated") 2024-09-02T12:10:26.097 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: SystemExit: Gateway process terminated 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:10:25] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.a) pid 19... 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: *** Caught signal (Aborted) ** 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: in thread 7f14e18035c0 thread_name:ceph-nvmeof-mon 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: ceph version 19.0.0-4996-g0ec90b1e (0ec90b1e61a7489b13d6d8432156a0417f35db7f) squid (dev) 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f14e2b956f0] 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 2: /lib64/libc.so.6(+0x8679a) [0x7f14e2bdd79a] 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 3: pthread_cond_wait() 2024-09-02T12:10:26.098 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f14e2f2d6b0] 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 5: (AsyncMessenger::wait()+0x7b) [0x7f14e3a1cceb] 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 6: (NVMeofGwMonitorClient::main(std::vector >)+0x18) [0x561c1ecfbb18] 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 7: main() 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 8: /lib64/libc.so.6(+0x29590) [0x7f14e2b80590] 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 9: __libc_start_main() 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: 10: _start() 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:10:25] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.a) pid 61... 2024-09-02T12:10:26.099 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:10:25] INFO server.py:125 (2): Stopping the server... 2024-09-02T12:10:26.100 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:10:25] INFO server.py:482 (2): Terminating discovery service... 2024-09-02T12:10:26.100 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:25 smithi064 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a[47519]: [02-Sep-2024 12:10:25] INFO state.py:545 (63): Cleanup OMAP on exit (discovery-smithi064) 2024-09-02T12:10:28.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.690 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.862 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:28.966 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:29.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:29.075 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:29.109 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:29.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:29.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~67s 2024-09-02T12:10:29.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~56s 2024-09-02T12:10:29.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~39s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~33s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~28s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~17s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~11s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~73s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~67s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~67s 2024-09-02T12:10:29.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~67s 2024-09-02T12:10:34.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.298 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.333 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.477 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.510 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.581 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:34.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~73s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~61s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~56s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~45s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~39s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~34s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~22s 2024-09-02T12:10:34.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~17s 2024-09-02T12:10:34.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~78s 2024-09-02T12:10:34.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~73s 2024-09-02T12:10:34.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~73s 2024-09-02T12:10:34.795 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~73s 2024-09-02T12:10:36.094 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:35 smithi064 bash[61962]: time="2024-09-02T12:10:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a in 10 seconds, resorting to SIGKILL" 2024-09-02T12:10:36.095 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:35 smithi064 podman[61962]: 2024-09-02 12:10:35.732899041 +0000 UTC m=+10.112263858 container died 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a, com.redhat.component=ubi9-container, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.openshift.wants=ceph,rbd, io.spdk.build.pkgdep-args=--rbd, summary=Ceph NVMe over Fabrics Gateway, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, distribution-scope=public, io.spdk.build.makeflags=, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.build.ceph-release=18.2.4, org.centos.version=9.0-21.el9, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.maintainer=Ceph Developers , io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, vcs-type=git, architecture=x86_64, version=1.2.17, url=https://github.com/ceph/ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.buildah.version=1.29.0, name=ceph-nvmeof, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.version=24.01, io.ceph.component=ceph-nvmeof, vendor=Red Hat, Inc., io.spdk.build.target-arch=, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.version=1.2.17, release=, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.nvmeof=None, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.git.branch=tags/1.2.17, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Ceph Developers , io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, build-date=2024-07-30 15:47:38 UTC) 2024-09-02T12:10:36.442 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 podman[61962]: 2024-09-02 12:10:36.207025861 +0000 UTC m=+10.586390660 container cleanup 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a, io.spdk.build.makeflags=, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.target-arch=, com.redhat.component=ubi9-container, io.ceph.maintainer=Ceph Developers , name=ceph-nvmeof, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.component=ceph-nvmeof, vcs-type=git, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers , io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, build-date=2024-07-30 15:47:38 UTC, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, distribution-scope=public, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.wants=ceph,rbd, url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.branch=undefined, io.buildah.version=1.29.0, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, architecture=x86_64, org.centos.version=9.0-21.el9, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, version=1.2.17, io.spdk.version=24.01, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.branch=tags/1.2.17, vendor=Red Hat, Inc., io.ceph.version=1.2.17, io.spdk.build.ceph-release=18.2.4, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, release=, io.ceph.nvmeof=None, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.spdk.build.pkgdep-args=--rbd, summary=Ceph NVMe over Fabrics Gateway) 2024-09-02T12:10:36.442 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 bash[61962]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a 2024-09-02T12:10:36.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 podman[62118]: 2024-09-02 12:10:36.441694633 +0000 UTC m=+0.707976325 container remove 5cc04673a2848b3090e7f765e961216e8d99fd48db50e8c2f20563c1d068ba0f (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-a, name=ceph-nvmeof, org.centos.version=9.0-21.el9, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, summary=Ceph NVMe over Fabrics Gateway, io.ceph.nvmeof=None, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.build.target-arch=, vcs-type=git, distribution-scope=public, maintainer=Ceph Developers , release=, io.spdk.git.repo=https://github.com/ceph/spdk.git, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.openshift.wants=ceph,rbd, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.buildah.version=1.29.0, io.ceph.component=ceph-nvmeof, architecture=x86_64, io.ceph.maintainer=Ceph Developers , io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, com.redhat.component=ubi9-container, io.spdk.build.makeflags=, build-date=2024-07-30 15:47:38 UTC, url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.branch=tags/1.2.17, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, vendor=Red Hat, Inc., io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.pkgdep-args=--rbd, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.version=1.2.17, io.spdk.build.ceph-release=18.2.4, version=1.2.17, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.version=24.01, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-02T12:10:36.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service: Main process exited, code=exited, status=137/n/a 2024-09-02T12:10:36.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service: Failed with result 'exit-code'. 2024-09-02T12:10:36.752 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 systemd[1]: Stopped Ceph nvmeof.nvmeof.a for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:10:36.753 INFO:journalctl@ceph.nvmeof.nvmeof.a.smithi064.stdout:Sep 02 12:10:36 smithi064 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.a.service: Consumed 9min 25.497s CPU time. 2024-09-02T12:10:37.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T12:10:37.611 INFO:teuthology.orchestra.run.smithi087.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:10:38.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 systemd[1]: Stopping Ceph nvmeof.nvmeof.b for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:10:39.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:10:38] ERROR server.py:111 (2): GatewayServer exception occurred: 2024-09-02T12:10:39.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: Traceback (most recent call last): 2024-09-02T12:10:39.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/src/control/__main__.py", line 39, in 2024-09-02T12:10:39.251 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: gateway.keep_alive() 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/src/control/server.py", line 526, in keep_alive 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: timedout = self.server.wait_for_termination(timeout=1) 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/src/__pypackages__/3.9/lib/grpc/_server.py", line 1118, in wait_for_termination 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: return _common.wait(self._state.termination_event.wait, 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 157, in wait 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: _wait_once(wait_fn, remaining, spin_cb) 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 112, in _wait_once 2024-09-02T12:10:39.252 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: wait_fn(timeout=timeout) 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/usr/lib64/python3.9/threading.py", line 581, in wait 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: signaled = self._cond.wait(timeout) 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/usr/lib64/python3.9/threading.py", line 316, in wait 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: gotit = waiter.acquire(True, timeout) 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: File "/src/control/__main__.py", line 17, in sigterm_handler 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: raise SystemExit(f"Gateway process terminated") 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: SystemExit: Gateway process terminated 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:10:38] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.b) pid 19... 2024-09-02T12:10:39.253 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: *** Caught signal (Aborted) ** 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: in thread 7f9f752565c0 thread_name:ceph-nvmeof-mon 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: ceph version 19.0.0-4996-g0ec90b1e (0ec90b1e61a7489b13d6d8432156a0417f35db7f) squid (dev) 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9f765e86f0] 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 2: /lib64/libc.so.6(+0x8679a) [0x7f9f7663079a] 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 3: pthread_cond_wait() 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f9f769806b0] 2024-09-02T12:10:39.254 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 5: (AsyncMessenger::wait()+0x7b) [0x7f9f7746fceb] 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 6: (NVMeofGwMonitorClient::main(std::vector >)+0x18) [0x560a4bde9b18] 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 7: main() 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 8: /lib64/libc.so.6(+0x29590) [0x7f9f765d3590] 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 9: __libc_start_main() 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: 10: _start() 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:10:38] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.b) pid 115... 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:10:38] INFO server.py:125 (2): Stopping the server... 2024-09-02T12:10:39.255 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:10:38] INFO server.py:482 (2): Terminating discovery service... 2024-09-02T12:10:39.256 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:38 smithi087 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b[41182]: [02-Sep-2024 12:10:38] INFO state.py:545 (117): Cleanup OMAP on exit (discovery-smithi087) 2024-09-02T12:10:39.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:39.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:39.912 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:39.953 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.030 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.106 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.136 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.207 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:40.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.347 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:40.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~78s 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~67s 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~61s 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~45s 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~39s 2024-09-02T12:10:40.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~28s 2024-09-02T12:10:40.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~22s 2024-09-02T12:10:40.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~84s 2024-09-02T12:10:40.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~78s 2024-09-02T12:10:40.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~78s 2024-09-02T12:10:40.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~78s 2024-09-02T12:10:45.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:45.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:45.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.564 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.724 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.826 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:45.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:45.970 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:46.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:46.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~84s 2024-09-02T12:10:46.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~73s 2024-09-02T12:10:46.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~67s 2024-09-02T12:10:46.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~56s 2024-09-02T12:10:46.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2024-09-02T12:10:46.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~45s 2024-09-02T12:10:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~34s 2024-09-02T12:10:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~28s 2024-09-02T12:10:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~89s 2024-09-02T12:10:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~84s 2024-09-02T12:10:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~84s 2024-09-02T12:10:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~84s 2024-09-02T12:10:49.134 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:48 smithi087 bash[52575]: time="2024-09-02T12:10:48Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b in 10 seconds, resorting to SIGKILL" 2024-09-02T12:10:49.134 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:48 smithi087 podman[52575]: 2024-09-02 12:10:48.862689815 +0000 UTC m=+10.138589687 container died 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.buildah.version=1.29.0, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.version=24.01, io.ceph.version=1.2.17, io.ceph.maintainer=Ceph Developers , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.git.branch=tags/1.2.17, distribution-scope=public, io.spdk.build.target-arch=, io.openshift.wants=ceph,rbd, io.spdk.build.pkgdep-args=--rbd, build-date=2024-07-30 15:47:38 UTC, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, com.redhat.component=ubi9-container, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.ceph-release=18.2.4, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.makeflags=, name=ceph-nvmeof, architecture=x86_64, org.centos.version=9.0-21.el9, url=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, io.ceph.component=ceph-nvmeof, vendor=Red Hat, Inc., io.ceph.nvmeof=None, maintainer=Ceph Developers , release=, version=1.2.17, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.git.repo=https://github.com/ceph/spdk.git, vcs-type=git, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440) 2024-09-02T12:10:49.492 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 podman[52575]: 2024-09-02 12:10:49.215691104 +0000 UTC m=+10.491590965 container cleanup 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b, io.ceph.maintainer=Ceph Developers , architecture=x86_64, io.spdk.build.makeflags=, io.ceph.version=1.2.17, com.redhat.component=ubi9-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.target-arch=, io.spdk.version=24.01, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.component=ceph-nvmeof, version=1.2.17, vendor=Red Hat, Inc., vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, summary=Ceph NVMe over Fabrics Gateway, name=ceph-nvmeof, url=https://github.com/ceph/ceph-nvmeof, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.openshift.wants=ceph,rbd, maintainer=Ceph Developers , io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, org.centos.version=9.0-21.el9, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.build.ceph-release=18.2.4, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.branch=undefined, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., build-date=2024-07-30 15:47:38 UTC, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.build.pkgdep-args=--rbd, release=, io.ceph.git.branch=tags/1.2.17, vcs-type=git, io.buildah.version=1.29.0, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.nvmeof=None, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440) 2024-09-02T12:10:49.493 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 bash[52575]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b 2024-09-02T12:10:49.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 podman[52732]: 2024-09-02 12:10:49.491623002 +0000 UTC m=+0.628060273 container remove 59b7589e0a00a3717a4bf1ecefbfe6ff8dbdfb56080250bf72a31c88630ed678 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-b, io.spdk.build.ceph-release=18.2.4, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, name=ceph-nvmeof, com.redhat.component=ubi9-container, distribution-scope=public, org.centos.version=9.0-21.el9, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, release=, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.version=1.2.17, io.spdk.build.pkgdep-args=--rbd, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.url=https://github.com/ceph/ceph-nvmeof, vendor=Red Hat, Inc., version=1.2.17, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.maintainer=Ceph Developers , io.spdk.build.makeflags=, io.spdk.build.target-arch=, build-date=2024-07-30 15:47:38 UTC, io.ceph.component=ceph-nvmeof, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.git.branch=tags/1.2.17, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, vcs-type=git, summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.branch=undefined, io.ceph.nvmeof=None, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, maintainer=Ceph Developers , architecture=x86_64, io.openshift.wants=ceph,rbd, io.buildah.version=1.29.0, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.version=24.01, url=https://github.com/ceph/ceph-nvmeof) 2024-09-02T12:10:49.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service: Main process exited, code=exited, status=137/n/a 2024-09-02T12:10:49.750 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service: Failed with result 'exit-code'. 2024-09-02T12:10:49.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 systemd[1]: Stopped Ceph nvmeof.nvmeof.b for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:10:49.751 INFO:journalctl@ceph.nvmeof.nvmeof.b.smithi087.stdout:Sep 02 12:10:49 smithi087 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.b.service: Consumed 9min 36.689s CPU time. 2024-09-02T12:10:50.293 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T12:10:50.485 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:10:51.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.147 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.259 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:51.372 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:51.434 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 systemd[1]: Stopping Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:10:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:51.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:51.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.560 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.592 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:51.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:51.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~89s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~78s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~73s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~62s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~56s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~50s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~39s 2024-09-02T12:10:51.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~34s 2024-09-02T12:10:51.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~95s 2024-09-02T12:10:51.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~89s 2024-09-02T12:10:51.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~89s 2024-09-02T12:10:51.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~89s 2024-09-02T12:10:51.847 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:10:51] ERROR server.py:111 (2): GatewayServer exception occurred: 2024-09-02T12:10:51.847 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: Traceback (most recent call last): 2024-09-02T12:10:51.847 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/src/control/__main__.py", line 39, in 2024-09-02T12:10:51.847 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: gateway.keep_alive() 2024-09-02T12:10:51.847 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/src/control/server.py", line 526, in keep_alive 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: timedout = self.server.wait_for_termination(timeout=1) 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/src/__pypackages__/3.9/lib/grpc/_server.py", line 1118, in wait_for_termination 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: return _common.wait(self._state.termination_event.wait, 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 157, in wait 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: _wait_once(wait_fn, remaining, spin_cb) 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/src/__pypackages__/3.9/lib/grpc/_common.py", line 112, in _wait_once 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: wait_fn(timeout=timeout) 2024-09-02T12:10:51.848 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/usr/lib64/python3.9/threading.py", line 581, in wait 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: signaled = self._cond.wait(timeout) 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/usr/lib64/python3.9/threading.py", line 316, in wait 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: gotit = waiter.acquire(True, timeout) 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: File "/src/control/__main__.py", line 17, in sigterm_handler 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: raise SystemExit(f"Gateway process terminated") 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: SystemExit: Gateway process terminated 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:10:51] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.c) pid 19... 2024-09-02T12:10:51.849 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: *** Caught signal (Aborted) ** 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: in thread 7f148e69d5c0 thread_name:ceph-nvmeof-mon 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: ceph version 19.0.0-4996-g0ec90b1e (0ec90b1e61a7489b13d6d8432156a0417f35db7f) squid (dev) 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f148fa2f6f0] 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 2: /lib64/libc.so.6(+0x8679a) [0x7f148fa7779a] 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 3: pthread_cond_wait() 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f148fdc76b0] 2024-09-02T12:10:51.850 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 5: (AsyncMessenger::wait()+0x7b) [0x7f14908b6ceb] 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 6: (NVMeofGwMonitorClient::main(std::vector >)+0x18) [0x55d6a0240b18] 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 7: main() 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 8: /lib64/libc.so.6(+0x29590) [0x7f148fa1a590] 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 9: __libc_start_main() 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: 10: _start() 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:10:51] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.c) pid 58... 2024-09-02T12:10:51.851 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:10:51] INFO server.py:125 (2): Stopping the server... 2024-09-02T12:10:51.852 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:10:51] INFO server.py:482 (2): Terminating discovery service... 2024-09-02T12:10:51.852 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:10:51 smithi094 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c[49537]: [02-Sep-2024 12:10:51] INFO state.py:545 (60): Cleanup OMAP on exit (discovery-smithi094) 2024-09-02T12:10:56.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:10:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:56.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:10:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:56.774 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:10:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:56.810 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:10:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:56.885 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:10:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:56.958 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:10:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:56.999 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:10:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:57.078 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:10:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:57.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:10:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:57.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:10:57.224 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:57.225 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:10:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:10:57.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:10:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:10:57.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~84s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~78s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~67s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~62s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~56s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~45s 2024-09-02T12:10:57.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~39s 2024-09-02T12:10:57.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~101s 2024-09-02T12:10:57.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2024-09-02T12:10:57.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2024-09-02T12:10:57.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~95s 2024-09-02T12:10:58.710 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: terminate called after throwing an instance of 'std::runtime_error' 2024-09-02T12:10:58.710 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: what(): Lost connection to the monitor (beacon timeout). 2024-09-02T12:10:58.710 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: *** Caught signal (Aborted) ** 2024-09-02T12:10:58.710 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: in thread 7fb966850640 thread_name:safe_timer 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: ceph version 19.0.0-4996-g0ec90b1e (0ec90b1e61a7489b13d6d8432156a0417f35db7f) squid (dev) 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb96fc4d6f0] 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb96fc9a94c] 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 3: raise() 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 4: abort() 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7fb96ffb1b21] 2024-09-02T12:10:58.711 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 6: /lib64/libstdc++.so.6(+0xad52c) [0x7fb96ffbd52c] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 7: /lib64/libstdc++.so.6(+0xad597) [0x7fb96ffbd597] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 8: /lib64/libstdc++.so.6(+0xad7f9) [0x7fb96ffbd7f9] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 9: /usr/bin/ceph-nvmeof-monitor-client(+0xa1342) [0x564a5cbc4342] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 10: (NVMeofGwMonitorClient::tick()+0x46) [0x564a5cc54d66] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 11: /usr/bin/ceph-nvmeof-monitor-client(+0x12123d) [0x564a5cc4423d] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 12: (CommonSafeTimer::timer_thread()+0x133) [0x7fb97091cd13] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 13: /usr/lib64/ceph/libceph-common.so.2(+0x27d731) [0x7fb97091f731] 2024-09-02T12:10:58.712 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 14: /lib64/libc.so.6(+0x89c02) [0x7fb96fc98c02] 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: 15: /lib64/libc.so.6(+0x10ec40) [0x7fb96fd1dc40] 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] ERROR server.py:42 (2): GatewayServer: SIGCHLD received signum=17 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] ERROR server.py:46 (2): PID of terminated child process is 19 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] ERROR server.py:111 (2): GatewayServer exception occurred: 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: Traceback (most recent call last): 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: File "/src/control/__main__.py", line 39, in 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: gateway.keep_alive() 2024-09-02T12:10:58.713 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: File "/src/control/server.py", line 530, in keep_alive 2024-09-02T12:10:58.714 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: time.sleep(self.spdk_ping_interval_in_seconds) 2024-09-02T12:10:58.714 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: File "/src/control/server.py", line 55, in sigchld_handler 2024-09-02T12:10:58.714 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: raise SystemExit(f"Gateway subprocess terminated {pid=} {exit_code=}") 2024-09-02T12:10:58.714 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: SystemExit: Gateway subprocess terminated pid=19 exit_code=-6 2024-09-02T12:10:58.714 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.d) pid 19... 2024-09-02T12:10:58.714 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] INFO server.py:448 (2): Aborting (client.nvmeof.nvmeof.d) pid 103... 2024-09-02T12:10:58.715 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] INFO server.py:125 (2): Stopping the server... 2024-09-02T12:10:58.715 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] INFO server.py:482 (2): Terminating discovery service... 2024-09-02T12:10:58.715 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:10:58 smithi099 ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d[22636]: [02-Sep-2024 12:10:58] INFO state.py:545 (105): Cleanup OMAP on exit (discovery-smithi099) 2024-09-02T12:11:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:01 smithi094 bash[53734]: time="2024-09-02T12:11:01Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c in 10 seconds, resorting to SIGKILL" 2024-09-02T12:11:01.891 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:01 smithi094 podman[53734]: 2024-09-02 12:11:01.625834964 +0000 UTC m=+10.137154496 container died 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.openshift.wants=ceph,rbd, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.version=1.2.17, io.ceph.nvmeof=None, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.ceph.maintainer=Ceph Developers , io.ceph.component=ceph-nvmeof, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, build-date=2024-07-30 15:47:38 UTC, io.spdk.build.ceph-release=18.2.4, io.spdk.build.target-arch=, distribution-scope=public, maintainer=Ceph Developers , architecture=x86_64, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=1.2.17, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.spdk.git.branch=undefined, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., com.redhat.component=ubi9-container, name=ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.repo=https://github.com/ceph/spdk.git, url=https://github.com/ceph/ceph-nvmeof, io.ceph.git.branch=tags/1.2.17, io.spdk.build.makeflags=, release=, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, summary=Ceph NVMe over Fabrics Gateway, org.centos.version=9.0-21.el9, vendor=Red Hat, Inc., io.buildah.version=1.29.0, io.spdk.version=24.01, vcs-type=git, io.spdk.build.pkgdep-args=--rbd, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba) 2024-09-02T12:11:02.259 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:01 smithi094 podman[53734]: 2024-09-02 12:11:01.998243157 +0000 UTC m=+10.509562689 container cleanup 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, name=ceph-nvmeof, io.ceph.version=1.2.17, io.ceph.nvmeof=None, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.openshift.wants=ceph,rbd, io.spdk.build.target-arch=, distribution-scope=public, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., vcs-type=git, version=1.2.17, io.ceph.component=ceph-nvmeof, io.spdk.build.makeflags=, architecture=x86_64, io.spdk.git.branch=undefined, build-date=2024-07-30 15:47:38 UTC, vendor=Red Hat, Inc., io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.branch=tags/1.2.17, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, summary=Ceph NVMe over Fabrics Gateway, url=https://github.com/ceph/ceph-nvmeof, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.buildah.version=1.29.0, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.ceph.maintainer=Ceph Developers , release=, io.spdk.version=24.01, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, org.centos.version=9.0-21.el9, io.spdk.build.ceph-release=18.2.4, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.build.pkgdep-args=--rbd, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, maintainer=Ceph Developers , com.redhat.component=ubi9-container, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF).) 2024-09-02T12:11:02.259 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:01 smithi094 bash[53734]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c 2024-09-02T12:11:02.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:11:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:11:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.408 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:11:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.444 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:11:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:11:02.566 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:02 smithi094 podman[53854]: 2024-09-02 12:11:02.257964358 +0000 UTC m=+0.632038591 container remove 64b1f23ef7632192f7977ba000040cf125a332261152d4c78da00258df43f8c6 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-c, io.spdk.git.repo=https://github.com/ceph/spdk.git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.version=1.2.17, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.branch=tags/1.2.17, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, org.centos.version=9.0-21.el9, io.ceph.component=ceph-nvmeof, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.maintainer=Ceph Developers , io.spdk.version=24.01, build-date=2024-07-30 15:47:38 UTC, io.spdk.git.branch=undefined, summary=Ceph NVMe over Fabrics Gateway, version=1.2.17, io.ceph.nvmeof=None, release=, url=https://github.com/ceph/ceph-nvmeof, maintainer=Ceph Developers , io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, vendor=Red Hat, Inc., io.ceph.summary=Ceph NVMe over Fabrics Gateway, distribution-scope=public, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., architecture=x86_64, name=ceph-nvmeof, io.ceph.url=https://github.com/ceph/ceph-nvmeof, com.redhat.component=ubi9-container, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.buildah.version=1.29.0, vcs-type=git, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.spdk.build.ceph-release=18.2.4, io.spdk.build.pkgdep-args=--rbd, io.openshift.wants=ceph,rbd, io.spdk.build.target-arch=, io.spdk.build.makeflags=) 2024-09-02T12:11:02.566 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:02 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Main process exited, code=exited, status=137/n/a 2024-09-02T12:11:02.567 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:02 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Failed with result 'exit-code'. 2024-09-02T12:11:02.567 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:02 smithi094 systemd[1]: Stopped Ceph nvmeof.nvmeof.c for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:11:02.567 INFO:journalctl@ceph.nvmeof.nvmeof.c.smithi094.stdout:Sep 02 12:11:02 smithi094 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.c.service: Consumed 2min 51.489s CPU time. 2024-09-02T12:11:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.590 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:11:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:11:02.656 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:11:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:11:02.732 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:11:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 3 2024-09-02T12:11:02.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:11:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:11:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:02.969 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:11:03.110 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T12:11:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:03.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:11:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:03.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~90s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~84s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~73s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~67s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~62s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~50s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~45s 2024-09-02T12:11:03.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~106s 2024-09-02T12:11:03.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~101s 2024-09-02T12:11:03.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~101s 2024-09-02T12:11:03.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~101s 2024-09-02T12:11:03.312 INFO:teuthology.orchestra.run.smithi099.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:11:03.459 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:03 smithi099 systemd[1]: Stopping Ceph nvmeof.nvmeof.d for e8e22764-6921-11ef-bcd4-c7b262605968... 2024-09-02T12:11:03.947 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:03 smithi099 podman[25352]: 2024-09-02 12:11:03.668475145 +0000 UTC m=+0.147737980 container died b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d, io.ceph.nvmeof=None, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.build.makeflags=, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.spdk.git.branch=undefined, distribution-scope=public, io.spdk.build.target-arch=, summary=Ceph NVMe over Fabrics Gateway, vendor=Red Hat, Inc., io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.build.ceph-release=18.2.4, io.ceph.url=https://github.com/ceph/ceph-nvmeof, org.centos.version=9.0-21.el9, io.spdk.version=24.01, architecture=x86_64, build-date=2024-07-30 15:47:38 UTC, io.spdk.build.pkgdep-args=--rbd, release=, com.redhat.component=ubi9-container, io.ceph.git.branch=tags/1.2.17, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.openshift.wants=ceph,rbd, version=1.2.17, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., name=ceph-nvmeof, url=https://github.com/ceph/ceph-nvmeof, vcs-type=git, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.buildah.version=1.29.0, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.component=ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, io.ceph.version=1.2.17, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, maintainer=Ceph Developers , io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.ceph.maintainer=Ceph Developers ) 2024-09-02T12:11:04.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:04 smithi099 podman[25352]: 2024-09-02 12:11:04.096137996 +0000 UTC m=+0.575400822 container cleanup b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d, io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, io.ceph.maintainer=Ceph Developers , io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, vcs-type=git, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, architecture=x86_64, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.spdk.git.branch=undefined, io.openshift.wants=ceph,rbd, maintainer=Ceph Developers , com.redhat.component=ubi9-container, io.ceph.url=https://github.com/ceph/ceph-nvmeof, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, build-date=2024-07-30 15:47:38 UTC, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.spdk.build.pkgdep-args=--rbd, io.spdk.version=24.01, io.spdk.build.ceph-release=18.2.4, summary=Ceph NVMe over Fabrics Gateway, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.build.target-arch=, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, vendor=Red Hat, Inc., io.ceph.git.branch=tags/1.2.17, release=, url=https://github.com/ceph/ceph-nvmeof, distribution-scope=public, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., version=1.2.17, io.buildah.version=1.29.0, io.ceph.nvmeof=None, io.ceph.component=ceph-nvmeof, org.centos.version=9.0-21.el9, io.ceph.version=1.2.17, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ceph-nvmeof, io.spdk.build.makeflags=) 2024-09-02T12:11:04.209 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:04 smithi099 bash[25352]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d 2024-09-02T12:11:04.675 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:04 smithi099 podman[25365]: 2024-09-02 12:11:04.305966329 +0000 UTC m=+0.631645803 container remove b0e3f86e16c78fb5803b18cbd7af3c8a07efcb3863b75580c46e3b7d0e04ce60 (image=quay.io/ceph/nvmeof:1.2, name=ceph-e8e22764-6921-11ef-bcd4-c7b262605968-nvmeof-nvmeof-d, io.ceph.nvmeof=None, vcs-ref=887c7841f275a0cbc00eddb8a038cde3935b95ba, io.spdk.build.pkgdep-args=--rbd, io.spdk.git.repo=https://github.com/ceph/spdk.git, io.spdk.build.target-arch=, io.ceph.version=1.2.17, io.ceph.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.git.repo=https://github.com/ceph/ceph-nvmeof, release=, url=https://github.com/ceph/ceph-nvmeof, vcs-type=git, name=ceph-nvmeof, com.redhat.component=ubi9-container, io.k8s.display-name=Ceph NVMe over Fabrics Gateway, io.ceph.url=https://github.com/ceph/ceph-nvmeof, org.centos.url=https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/, version=1.2.17, io.spdk.build.configure-args=--with-rbd --disable-tests --disable-unit-tests --disable-examples --enable-debug, io.spdk.git.commit=a16bb032516da05ea2b7c38fd0ad18e8a7190440, io.spdk.build.makeflags=, architecture=x86_64, io.ceph.git.branch=tags/1.2.17, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.spdk.version=24.01, io.ceph.summary=Ceph NVMe over Fabrics Gateway, io.ceph.component=ceph-nvmeof, description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.tags=ceph,nvme-of,nvme-of gateway,rbd,block storage, io.k8s.description=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.ceph.maintainer=Ceph Developers , summary=Ceph NVMe over Fabrics Gateway, io.openshift.wants=ceph,rbd, io.spdk.build.ceph-release=18.2.4, io.spdk.git.branch=undefined, io.openshift.expose-services=4420/tcp:nvme,5500/tcp:grpc,8009/tcp:nvme-disc,10008/tcp:prom, io.buildah.version=1.29.0, distribution-scope=public, build-date=2024-07-30 15:47:38 UTC, maintainer=Ceph Developers , org.centos.version=9.0-21.el9, vendor=Red Hat, Inc., io.ceph.git.commit=887c7841f275a0cbc00eddb8a038cde3935b95ba) 2024-09-02T12:11:04.676 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:04 smithi099 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service: Deactivated successfully. 2024-09-02T12:11:04.676 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:04 smithi099 systemd[1]: Stopped Ceph nvmeof.nvmeof.d for e8e22764-6921-11ef-bcd4-c7b262605968. 2024-09-02T12:11:04.676 INFO:journalctl@ceph.nvmeof.nvmeof.d.smithi099.stdout:Sep 02 12:11:04 smithi099 systemd[1]: ceph-e8e22764-6921-11ef-bcd4-c7b262605968@nvmeof.nvmeof.d.service: Consumed 9min 53.851s CPU time. 2024-09-02T12:11:05.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:11:05.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:11:05.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:11:05.241 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:11:05.272 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T12:11:05.299 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-02T12:11:05.301 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi038/crash 2024-09-02T12:11:05.302 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T12:11:05.343 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T12:11:05.343 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-09-02T12:11:05.345 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi064/crash 2024-09-02T12:11:05.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T12:11:05.389 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T12:11:05.390 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-09-02T12:11:05.391 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi087/crash 2024-09-02T12:11:05.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T12:11:05.502 INFO:teuthology.orchestra.run.smithi087.stderr:tar: /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T12:11:05.502 INFO:teuthology.orchestra.run.smithi087.stderr:tar: Error is not recoverable: exiting now 2024-09-02T12:11:05.504 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi094/crash 2024-09-02T12:11:05.505 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T12:11:05.553 INFO:teuthology.orchestra.run.smithi094.stderr:tar: /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T12:11:05.553 INFO:teuthology.orchestra.run.smithi094.stderr:tar: Error is not recoverable: exiting now 2024-09-02T12:11:05.555 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi099/crash 2024-09-02T12:11:05.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T12:11:05.636 INFO:teuthology.orchestra.run.smithi099.stderr:tar: /var/lib/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T12:11:05.636 INFO:teuthology.orchestra.run.smithi099.stderr:tar: Error is not recoverable: exiting now 2024-09-02T12:11:05.637 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-02T12:11:05.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'is in error state' | egrep -v 'failed cephadm daemon' | head -n 1 2024-09-02T12:11:05.670 INFO:tasks.cephadm:Compressing logs... 2024-09-02T12:11:05.670 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T12:11:05.673 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T12:11:05.700 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-09-02T12:11:05.700 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T12:11:05.700 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T12:11:05.701 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 81.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T12:11:05.701 INFO:teuthology.orchestra.run.smithi038.stderr: 90.0% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T12:11:05.703 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-02T12:11:05.703 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m0.015s 2024-09-02T12:11:05.703 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m0.012s 2024-09-02T12:11:05.703 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.016s 2024-09-02T12:11:05.713 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T12:11:05.715 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T12:11:05.718 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T12:11:05.737 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.0.log 2024-09-02T12:11:05.738 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-09-02T12:11:05.738 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.1.log 2024-09-02T12:11:05.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T12:11:05.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log 2024-09-02T12:11:05.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-09-02T12:11:05.740 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose --/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log: /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.a.log 2024-09-02T12:11:05.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log 2024-09-02T12:11:05.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.a.log: 82.8% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-09-02T12:11:05.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mgr.x.log 2024-09-02T12:11:05.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T12:11:05.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mgr.x.log: /var/log/ceph/cephadm.log: 90.1% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-09-02T12:11:05.742 INFO:teuthology.orchestra.run.smithi064.stderr: 88.1% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log.gz 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:: No such file or directory 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.4.log 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log: --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.2.log 2024-09-02T12:11:05.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log 2024-09-02T12:11:05.745 INFO:teuthology.orchestra.run.smithi087.stderr: 81.8% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-09-02T12:11:05.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.b.log 2024-09-02T12:11:05.747 INFO:teuthology.orchestra.run.smithi064.stderr: 91.2% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T12:11:05.747 INFO:teuthology.orchestra.run.smithi064.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T12:11:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T12:11:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:: No such file or directory 2024-09-02T12:11:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T12:11:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 84.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T12:11:05.750 INFO:teuthology.orchestra.run.smithi099.stderr: 90.1% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T12:11:05.751 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-09-02T12:11:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m0.012s 2024-09-02T12:11:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:user 0m0.013s 2024-09-02T12:11:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m0.010s 2024-09-02T12:11:05.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.3.log 2024-09-02T12:11:05.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T12:11:05.752 INFO:teuthology.orchestra.run.smithi087.stderr: 90.2% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-09-02T12:11:05.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.3.log: 88.2% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log.gz/var/log/ceph/cephadm.log: 2024-09-02T12:11:05.752 INFO:teuthology.orchestra.run.smithi087.stderr: 89.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T12:11:05.752 INFO:teuthology.orchestra.run.smithi087.stderr: 91.2% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.7.log 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:: No such file or directory 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.c.log 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-09-02T12:11:05.754 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.6.log 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.5.log 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr: 81.7% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr: 90.3% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log: /var/log/ceph/cephadm.log: 89.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr: 88.2% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph.log.gz 2024-09-02T12:11:05.755 INFO:teuthology.orchestra.run.smithi094.stderr: 90.9% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T12:11:05.849 INFO:teuthology.orchestra.run.smithi064.stderr: 90.6% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mgr.x.log.gz 2024-09-02T12:11:05.897 INFO:teuthology.orchestra.run.smithi094.stderr: 92.7% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.c.log.gz 2024-09-02T12:11:05.914 INFO:teuthology.orchestra.run.smithi087.stderr: 92.5% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.b.log.gz 2024-09-02T12:11:05.964 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.0.log.gz 2024-09-02T12:11:06.000 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.4.log.gz 2024-09-02T12:11:06.073 INFO:teuthology.orchestra.run.smithi094.stderr: 93.9% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.6.log.gz 2024-09-02T12:11:06.100 INFO:teuthology.orchestra.run.smithi094.stderr: 93.7% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.5.log.gz 2024-09-02T12:11:06.128 INFO:teuthology.orchestra.run.smithi087.stderr: 93.9% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.2.log.gz 2024-09-02T12:11:06.263 INFO:teuthology.orchestra.run.smithi064.stderr: 90.7% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-mon.a.log.gz 2024-09-02T12:11:06.304 INFO:teuthology.orchestra.run.smithi064.stderr: 94.3% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.1.log.gz 2024-09-02T12:11:06.306 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-02T12:11:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.580s 2024-09-02T12:11:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m1.403s 2024-09-02T12:11:06.306 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.064s 2024-09-02T12:11:06.404 INFO:teuthology.orchestra.run.smithi094.stderr: 94.5% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.7.log.gz 2024-09-02T12:11:06.406 INFO:teuthology.orchestra.run.smithi094.stderr: 2024-09-02T12:11:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m0.673s 2024-09-02T12:11:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m1.453s 2024-09-02T12:11:06.406 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m0.078s 2024-09-02T12:11:06.680 INFO:teuthology.orchestra.run.smithi087.stderr: 95.0% -- replaced with /var/log/ceph/e8e22764-6921-11ef-bcd4-c7b262605968/ceph-osd.3.log.gz 2024-09-02T12:11:06.682 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-02T12:11:06.682 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m0.952s 2024-09-02T12:11:06.682 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m1.690s 2024-09-02T12:11:06.682 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.098s 2024-09-02T12:11:06.682 INFO:tasks.cephadm:Archiving logs... 2024-09-02T12:11:06.683 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi038/log 2024-09-02T12:11:06.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T12:11:06.713 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi064/log 2024-09-02T12:11:06.714 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T12:11:06.974 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi087/log 2024-09-02T12:11:06.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T12:11:07.251 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi094/log 2024-09-02T12:11:07.252 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T12:11:07.514 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi099/log 2024-09-02T12:11:07.515 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T12:11:07.549 INFO:tasks.cephadm:Removing cluster... 2024-09-02T12:11:07.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force 2024-09-02T12:11:07.758 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:11:08.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force 2024-09-02T12:11:08.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.0 2024-09-02T12:11:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.1 2024-09-02T12:11:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.335 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.2 2024-09-02T12:11:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.3 2024-09-02T12:11:08.381 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:11:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.445 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.4 2024-09-02T12:11:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.518 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.5 2024-09-02T12:11:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.555 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.6 2024-09-02T12:11:08.628 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@osd.7 2024-09-02T12:11:08.638 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force 2024-09-02T12:11:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.a 2024-09-02T12:11:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.733 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.b 2024-09-02T12:11:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mon.c 2024-09-02T12:11:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl status ceph-e8e22764-6921-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T12:11:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 4 2024-09-02T12:11:08.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~107s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~96s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~79s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~73s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~68s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~56s 2024-09-02T12:11:08.828 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~51s 2024-09-02T12:11:08.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~112s 2024-09-02T12:11:08.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~107s 2024-09-02T12:11:08.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~107s 2024-09-02T12:11:08.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~107s 2024-09-02T12:11:08.835 INFO:teuthology.orchestra.run.smithi087.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:11:09.091 DEBUG:teuthology.orchestra.run.smithi094:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force 2024-09-02T12:11:09.289 INFO:teuthology.orchestra.run.smithi094.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:11:09.552 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e8e22764-6921-11ef-bcd4-c7b262605968 --force 2024-09-02T12:11:09.753 INFO:teuthology.orchestra.run.smithi099.stdout:Deleting cluster with fsid: e8e22764-6921-11ef-bcd4-c7b262605968 2024-09-02T12:11:10.009 INFO:tasks.cephadm:Removing cephadm ... 2024-09-02T12:11:10.010 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T12:11:10.029 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T12:11:10.048 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T12:11:10.067 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T12:11:10.085 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T12:11:10.100 INFO:tasks.cephadm:Teardown complete 2024-09-02T12:11:10.101 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-02T12:11:10.111 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-02T12:11:10.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T12:11:10.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T12:11:10.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T12:11:10.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T12:11:10.127 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T12:11:10.332 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-02T12:11:10.342 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-02T12:11:10.342 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T12:11:10.345 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T12:11:10.348 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T12:11:10.351 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T12:11:10.353 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T12:11:10.360 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-09-02T12:11:10.362 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-02T12:11:10.365 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T12:11:10.365 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-09-02T12:11:10.365 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 -38us[ -36us] +/- 38ms 2024-09-02T12:11:10.365 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -215us[ -215us] +/- 68ms 2024-09-02T12:11:10.365 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +359us[ +359us] +/- 65ms 2024-09-02T12:11:10.365 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T12:11:10.366 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-09-02T12:11:10.367 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T12:11:10.367 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-02T12:11:10.368 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 -62us[ -66us] +/- 38ms 2024-09-02T12:11:10.368 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -230us[ -230us] +/- 68ms 2024-09-02T12:11:10.368 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +337us[ +337us] +/- 65ms 2024-09-02T12:11:10.368 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T12:11:10.369 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 1: ntpq: command not found 2024-09-02T12:11:10.369 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-09-02T12:11:10.370 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T12:11:10.370 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-09-02T12:11:10.371 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 -58us[ -64us] +/- 38ms 2024-09-02T12:11:10.371 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 12 -231us[ -236us] +/- 68ms 2024-09-02T12:11:10.371 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +315us[ +315us] +/- 65ms 2024-09-02T12:11:10.371 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T12:11:10.372 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T12:11:10.373 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-09-02T12:11:10.373 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 -72us[ -98us] +/- 38ms 2024-09-02T12:11:10.373 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -233us[ -233us] +/- 68ms 2024-09-02T12:11:10.373 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +330us[ +330us] +/- 65ms 2024-09-02T12:11:10.373 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T12:11:10.374 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T12:11:10.374 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2024-09-02T12:11:10.374 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv01.front.sepia.ceph.com 3 7 377 8 -45us[ -51us] +/- 38ms 2024-09-02T12:11:10.374 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 11 -188us[ -195us] +/- 68ms 2024-09-02T12:11:10.375 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +315us[ +308us] +/- 65ms 2024-09-02T12:11:10.375 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T12:11:10.375 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-02T12:11:10.384 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-02T12:11:10.385 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-02T12:11:10.414 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T12:11:10.494 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T12:11:10.568 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T12:11:10.637 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T12:11:10.708 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T12:11:10.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T12:11:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:10.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T12:11:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:10.823 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T12:11:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:10.856 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T12:11:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:10.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-02T12:11:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:10.915 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-02T12:11:10.925 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-02T12:11:10.965 INFO:teuthology.task.internal:Duration was 1359.157888 seconds 2024-09-02T12:11:10.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-02T12:11:10.998 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-02T12:11:10.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T12:11:11.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T12:11:11.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T12:11:11.006 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T12:11:11.008 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T12:11:11.044 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T12:11:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T12:11:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T12:11:11.049 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T12:11:11.052 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T12:11:11.499 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-02T12:11:11.499 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-09-02T12:11:11.499 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T12:11:11.530 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-09-02T12:11:11.531 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T12:11:11.555 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-09-02T12:11:11.556 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T12:11:11.591 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2024-09-02T12:11:11.591 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T12:11:11.630 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-09-02T12:11:11.630 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T12:11:11.660 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-02T12:11:11.660 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T12:11:11.663 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T12:11:11.665 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T12:11:11.668 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T12:11:11.673 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T12:11:11.728 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-02T12:11:11.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T12:11:11.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T12:11:11.737 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T12:11:11.739 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T12:11:11.743 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T12:11:11.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-02T12:11:11.959 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-02T12:11:11.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T12:11:11.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T12:11:12.008 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T12:11:12.033 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T12:11:12.059 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T12:11:12.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-02T12:11:12.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.099 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.101 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.104 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.118 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-09-02T12:11:12.120 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-09-02T12:11:12.123 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-09-02T12:11:12.124 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2024-09-02T12:11:12.147 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-09-02T12:11:12.170 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:12.185 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:12.206 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:12.221 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:12.235 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T12:11:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:12.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-02T12:11:12.260 INFO:teuthology.task.internal:Transferring archived files... 2024-09-02T12:11:12.261 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi038 2024-09-02T12:11:12.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T12:11:12.299 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi064 2024-09-02T12:11:12.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T12:11:12.336 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi087 2024-09-02T12:11:12.336 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T12:11:12.369 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi094 2024-09-02T12:11:12.369 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T12:11:12.402 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vallariag-2024-09-02_11:40:02-nvmeof-main-distro-default-smithi/7886689/remote/smithi099 2024-09-02T12:11:12.404 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T12:11:12.434 INFO:teuthology.task.internal:Removing archive directory... 2024-09-02T12:11:12.435 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T12:11:12.437 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T12:11:12.439 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T12:11:12.442 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T12:11:12.444 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T12:11:12.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-02T12:11:12.499 INFO:teuthology.task.internal:Not uploading archives. 2024-09-02T12:11:12.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-02T12:11:12.523 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-02T12:11:12.523 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T12:11:12.525 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T12:11:12.527 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T12:11:12.529 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T12:11:12.532 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T12:11:12.538 INFO:teuthology.orchestra.run.smithi038.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 12:11 /home/ubuntu/cephtest 2024-09-02T12:11:12.540 INFO:teuthology.orchestra.run.smithi064.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 12:11 /home/ubuntu/cephtest 2024-09-02T12:11:12.542 INFO:teuthology.orchestra.run.smithi087.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 12:11 /home/ubuntu/cephtest 2024-09-02T12:11:12.544 INFO:teuthology.orchestra.run.smithi094.stdout: 262217 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 2 12:11 /home/ubuntu/cephtest 2024-09-02T12:11:12.544 INFO:teuthology.orchestra.run.smithi094.stdout: 556392 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 2 12:06 /home/ubuntu/cephtest/mnt.3 2024-09-02T12:11:12.545 INFO:teuthology.orchestra.run.smithi094.stdout: 556393 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 2 12:06 /home/ubuntu/cephtest/mnt.3/client.3 2024-09-02T12:11:12.545 INFO:teuthology.orchestra.run.smithi094.stdout: 670144 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 12:06 /home/ubuntu/cephtest/mnt.3/client.3/tmp 2024-09-02T12:11:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T12:11:12.546 INFO:teuthology.orchestra.run.smithi094.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-02T12:11:12.546 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-02T12:11:12.553 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-02T12:11:12.564 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-02T12:11:12.599 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi094 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-02T12:11:12.603 INFO:teuthology.run:Summary data: description: nvmeof/thrash/{base/install centos_latest clusters/4-gateways-1-initiator conf/{disable-pool-app} gateway-initiator-setup/3-subsys-60-namespace thrashers/nvmeof_thrash workloads/fio} duration: 1359.1578884124756 failure_reason: 'Command failed (workunit test nvmeof/fio_test.sh) on smithi094 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6bdd3521695a0621868ce5034eaba1fd0b6e3cd1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3 CEPH_MNT=/home/ubuntu/cephtest/mnt.3 IOSTAT_INTERVAL=10 RBD_POOL=mypool RUNTIME=600 timeout 30m /home/ubuntu/cephtest/clone.client.3/qa/workunits/nvmeof/fio_test.sh --rbd_iostat''' flavor: default owner: scheduled_vallariag@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=045a301960eb41d593a650e510901aa0 status: fail success: false 2024-09-02T12:11:12.603 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-02T12:11:12.626 INFO:teuthology.orchestra.run.smithi099.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 12:11 /home/ubuntu/cephtest 2024-09-02T12:11:12.708 INFO:teuthology.run:FAIL